85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is an IVF index?

An IVF (inverted file) index is an approximate nearest neighbour index that clusters vectors around centroids and, at query time, scans only the clusters nearest the query instead of every vector.

Checked

An IVF index divides vector space into cells, each around a centroid learned with k-means. Every stored vector is filed under its nearest centroid. A query is compared to the centroids first, then only to vectors in the closest few cells.

The idea is old by AI standards. Hervé Jégou, Matthijs Douze and Cordelia Schmid paired an inverted file with product quantization in Product quantization for nearest neighbor search (IEEE TPAMI, 2011), validated on two billion vectors. Meta's Faiss library implements IVF indexes, and pgvector ships a plain version as IVFFlat.

How lists and probes work

  • Training

    The index must learn its centroids from sample data before vectors are added. pgvector's README says to create an IVFFlat index after the table has some data.

  • Lists (nlist)

    The number of clusters. pgvector suggests starting at rows / 1000 for up to 1M rows and sqrt(rows) above that.

  • Probes (nprobe)

    How many clusters each query scans. Faiss defaults to 1; pgvector suggests starting at sqrt(lists). More probes raise recall and cost speed, and probing every list equals brute force.

Variants in use

  • IVFFlat

    Stores full vectors in each list. Simple and accurate within the probed clusters.

  • IVF with quantization

    Stores compressed codes instead of full vectors, as in IVF-PQ. Pinecone's pg_pinecone (2026-10-27) runs an IVF plus RaBitQ engine inside Postgres and, in Pinecone's own 1M by 768-dimension benchmark, reports a 136 MB index against 2.0 GB for a halfvec HNSW index.

  • Object-storage engines

    turbopuffer's architecture docs explain that each object storage round trip takes about 100 ms, so its vector index is centroid-based to cut round trips and write amplification compared with graph indexes.

Where IVF fails, and what it is confused with

  • Boundary misses

    A true neighbour that sits just across a cell boundary is missed unless its cell is probed. Low probe counts make recall uneven across queries.

  • Stale centroids

    Centroids reflect the data at build time. When the distribution shifts, for example a new product line or language, clusters become unbalanced and recall drifts until the index is rebuilt.

  • Speed and recall versus HNSW

    pgvector's docs say IVFFlat builds faster and uses less memory than HNSW, but has a weaker speed and recall trade-off.

  • Not the full-text inverted index

    A search engine's inverted index maps each word to the documents containing it. An IVF vector index maps each centroid to its vectors. The name is shared; the contents differ.

How Motiontic's engineers apply it

Motiontic's engineers use IVF where memory or storage cost matters more than the last points of recall, then tune probes against recall@k on the client's queries. Rebuild triggers are written into the runbook, so a shift in the data does not silently erode retrieval quality.

IVF compared with HNSW

IVF compared with HNSW
IVFHNSW
StructureClusters around trained centroidsLayered proximity graph
Needs trainingYes, before indexingNo
Build timeFasterSlower
MemoryLowerHigher
Main query knobProbes (nprobe)ef_search
Speed and recall balanceWeaker at the same speedStronger
SuitsLarge, cost-sensitive, object-storage indexesLatency-sensitive in-memory search

Common questions

IVFFlat or HNSW in pgvector?

pgvector's README says HNSW has the better speed and recall trade-off, while IVFFlat builds faster and uses less memory. IVFFlat also needs data in the table before the index is created.

How many lists should an IVFFlat index have?

pgvector suggests starting with rows / 1000 for tables up to 1M rows and sqrt(rows) beyond that, then adjusting against measured recall.

What does nprobe do?

It sets how many clusters a query searches. Raising it improves recall and slows the query; setting it to the total number of clusters gives the same results as brute force.

Is an IVF index the same as an inverted index in a search engine?

No. A full-text inverted index lists documents per word. An IVF vector index lists vectors per cluster centroid. Both are lookup tables, but they index different things.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.