This means the difference between two embeddings is itself a direction with a meaning, the mean of several is a concept, and projecting one onto another keeps or removes one component of what a text is about. Most systems hide embeddings behind a similarity API and throw that structure away. In Scry a vector is a first-class value: mint one from any text, then add, subtract, and project in SQL, and rank a whole corpus along the result. Text search speaks a full operator grammar — exact phrases, exclusion, OR, regex, fuzzy, proximity — and scry_lex carries that whole grammar into SQL as one predicate, compiled server-side to each relation's own text indexes. Everything below runs as written against the live schema.
Which way has LessWrong drifted, doom or optimism?
Mint the two poles, take the balanced axis between them, and average every chunk's projection quarter by quarter, joined to real post timestamps. One query returns the emotional trajectory of a community.
POST /v1/scry/embed {"text": "we will solve alignment", "name": "optimism"}
POST /v1/scry/embed {"text": "we are not going to make it", "name": "doom"}
SELECT toStartOfQuarter(p.original_timestamp) AS quarter,
avg(scry_cosine_similarity(e.embedding,
scry_contrast_axis_balanced(@optimism, @doom))) AS lean,
uniq(e.target_key) AS posts
FROM embeddings.chunks AS e
JOIN forums.posts AS p ON p.post_key = e.target_key
WHERE e.source = 'forum_posts' AND e.model_name = 'voyage-4-lite'
AND p.source = 'lesswrong'
GROUP BY quarter
ORDER BY quarter
LIMIT 100; How entangled are two ideas, and what is nearest to one?
Measure how much of one idea lives inside another, then rank a corpus by distance to it. scry_debias_vector subtracts the shared component when you want the residue instead.
SELECT scry_cosine_similarity(@alignment, @safety) AS entanglement,
scry_debias_removed_fraction(@alignment, @safety) AS removed_fraction
LIMIT 1;
SELECT post_key, chunk_index,
scry_vector_topk_distance(embedding_voyage4, @alignment) AS dist
FROM embeddings.forum_posts
WHERE model_name = 'voyage-4-lite' AND post_key LIKE 'lesswrong%'
ORDER BY dist ASC
LIMIT 10; How fast is an idea catching on — minus its noise?
The search grammar rides inside SQL as one predicate: exact phrase, exclusion, OR, regex. scry_lex compiles server-side to the relation's own token indexes, so the rest is ordinary SQL — count by month and the adoption curve falls out.
SELECT toStartOfMonth(original_timestamp) AS month, count() AS hits
FROM hackernews.items
WHERE scry_lex('"scaling laws" -crypto')
AND original_timestamp >= '2018-01-01'
GROUP BY month
ORDER BY month
LIMIT 120; What does a community actually call the thing?
A regex needs only a short literal run to prune by index before it scans. Match every model name in three months of Reddit comments, extract, and group — a vocabulary census, not a sample.
SELECT arrayJoin(extractAll(lower(body), 'gpt-[0-9]{1,2}(?:\.[0-9])?[a-z]*')) AS model,
count() AS hits
FROM reddit.comments
WHERE scry_lex('/GPT-[0-9]/')
AND created_utc >= now() - INTERVAL 90 DAY
GROUP BY model
ORDER BY hits DESC
LIMIT 10; Where do two ideas touch?
NEAR/60 matches both orders within sixty characters — tighter than co-occurrence in a document, looser than a phrase. Slop ("exact phrase"~3) and typo tolerance (word~1) live in the same grammar.
SELECT created_utc, subreddit, author, leftUTF8(body, 120) AS excerpt
FROM reddit.comments
WHERE scry_lex('lithium NEAR/60 sodium')
AND created_utc >= now() - INTERVAL 30 DAY
ORDER BY created_utc DESC
LIMIT 5; Which corners of the archive even talk about it?
One call compiles the same line against every relation with a text plane and counts each — the phrase's distribution across the whole estate, denominators included, before you spend a single scan.
POST /v1/scry/compile
{"q": "\"scaling laws\" -crypto", "relation": "*", "counts": true}
→ 31 relations, counted: openalex.works · academic.catalog · twitter
· forums · hackernews · bluesky · mastodon · github · … When exactly did this market change its mind?
Every Manifold bet carries the market probability before and after it. That gives the full price path of a question, live from the source.
SELECT created_at_source, prob_before, prob_after, amount, outcome
FROM manifold.bets
WHERE contract_id = '<market-id>'
AND is_redemption = 0
ORDER BY created_at_source
LIMIT 1000;
Who used the phrase first, and in which room?
Token-indexed lexical search covers the Reddit archive end to end — and completeness is measured against Reddit's own ID counters and published per relation, so you know when an empty result means absence rather than a gap. Scope first, match second, and every matching row comes back with its provenance.
SELECT id, subreddit, author, created_utc, score, body
FROM reddit.comments
WHERE subreddit = 'Physics'
AND hasAllTokens(search_text_lc, ['room', 'temperature', 'superconductor'])
ORDER BY created_utc ASC
LIMIT 50;
Text
scry_lex('"exact phrase" -noise /regex/')the whole search grammar as one SQL predicate
a NEAR/50 b · "phrase"~3 · word~1proximity, slop, and typo tolerance, index-pruned
hasAllTokens(search_text_lc, […])token-indexed match over full archives — every row, not a ranked sample
toStartOfMonth · uniq · argMintime series, distinct authors, first observations
JOIN … ON post_key / hn_id / contract_idsource-native keys across relations
Meaning
POST /v1/scry/embed → @handlemint a named vector from any text
scry_contrast_axis(@a, @b)the axis between two ideas
scry_project_onto · scry_debias_vectorkeep, or remove, one component of meaning
scry_seed_centroid([@x, @y, …])a concept built from examples
scry_cosine_similarity · scry_handle_matrixmeasure one pair, or all pairs at once
scry_vector_topk_distanceANN-rank an indexed relation along any of the above