Vector and hybrid search

The problem: "sneakers" finds nothing

An agency searches its links for sneakers for kids. Its links are titled "Children's running shoes", "Trainers for school", "Junior footwear sale". BM25 finds none of them: not one word matches. The user meant something the words don't share. To search by meaning, Snip needs a different kind of index.

Embeddings

An embedding model turns a piece of text into a list of numbers, a vector, typically a few hundred to a few thousand numbers long, such that texts with similar meanings get vectors that point in similar directions. "Sneakers for kids" and "children's running shoes" land close together; "quarterly tax filing" lands far away.

Search then becomes geometry: embed the query, and find the stored vectors nearest to it, usually by cosine similarity (the angle between vectors) or inner product.

const cosine = (a: number[], b: number[]) => {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    na += a[i] * a[i];
    nb += b[i] * b[i];
  }
  return dot / Math.sqrt(na * nb);
};

Exact search doesn't scale

The exact answer compares the query with every stored vector. For Snip's 600 million links at 768 numbers each, that's about 460 billion multiplications per search. Far too slow for a search box.

Approximate nearest neighbour (ANN) indexes trade a little accuracy for enormous speed: they return most of the true nearest neighbours (the fraction is called recall) while looking at a tiny fraction of the vectors.

• HNSW (Hierarchical Navigable Small World) builds a graph where each vector links to its near neighbours, in several layers: sparse upper layers with long links for travelling far quickly, denser lower layers for the final approach, much like a skip list. A search greedily walks the graph towards the query. Its authors show search cost growing logarithmically with the number of vectors.
• IVF (inverted file) clusters the vectors and, at query time, searches only the few clusters nearest the query.

pgvector, the Postgres extension, offers both. Its README sums up the trade-off: HNSW has better query performance for the speed-recall trade-off, but slower builds and more memory; IVFFlat builds faster and uses less memory, but queries less well. Both have a knob that buys recall with time (hnsw.ef_search, ivfflat.probes).

CREATE EXTENSION vector;
ALTER TABLE links ADD COLUMN embedding vector(768);
CREATE INDEX ON links USING hnsw (embedding vector_cosine_ops);

-- <=> is cosine distance: the 10 nearest titles to the query's embedding
SELECT code, title FROM links ORDER BY embedding <=> $1 LIMIT 10;

The filtering trap

Snip's searches are always WHERE owner_id = 4821. With an approximate index, the filter is applied after the index returns its candidates. pgvector's README gives the arithmetic: if a filter matches 10% of rows, then with HNSW's default of 40 candidates, only about 4 rows survive on average, so a "top 10" query can return 4. Fixes: raise the candidate count, use iterative scans that keep searching until enough rows pass (pgvector 0.8.0 added these), or partition the vectors so each owner's search covers only their own (like routing in 10.04 Sharding a search cluster).

Hybrid search: words and meaning together

Vectors find "trainers" for "sneakers", but they're vague with exact things: a campaign code like SPR-2026-DE, a product name, a rare word. BM25 is excellent at those. Hybrid search runs both and merges the results.

Merging is awkward because BM25 scores and cosine similarities are on unrelated scales. Reciprocal rank fusion (RRF) ignores the scores and uses only ranks: each list gives a document 1 / (k + rank), and the totals decide the order.

/** Combine ranked lists of IDs: each list contributes 1 / (k + rank) for every document it contains. */
export function rrf(lists: string[][], k = 60): [string, number][] {
  const scores = new Map<string, number>();
  for (const list of lists) list.forEach((id, i) => scores.set(id, (scores.get(id) ?? 0) + 1 / (k + i + 1)));
  return [...scores].sort((a, b) => b[1] - a[1]);
}

Tested with a keyword list a, b, c, d and a vector list c, a, e: the fused order was a (0.0325), c (0.0323), b, e, d. Documents that both methods rank well rise to the top, and a document only one method found still appears. The constant k = 60 comes from the original paper, where it was near-optimal but not critical; Elasticsearch uses 60 as its default too.

What it costs

• Embedding everything, and re-embedding everything when you change models: vectors from different models can't be compared.
• Memory: 600 million vectors of 768 four-byte numbers is about 1.8 TB before the graph's links.
• Recall is below 100%: measure it on real queries against exact search, and tune the candidate count.
• Explainability: "why did this match?" has no word to highlight.

Snip starts small: embeddings for titles and notes, an HNSW index per large agency, BM25 for everyone, and RRF to merge.

Recap

Interview questions