85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is ColPali?

ColPali is a document retrieval model that embeds images of document pages with a vision language model, producing many vectors per page that are matched to query tokens by late interaction, with no OCR or text extraction step.

Checked

Most RAG pipelines start by turning PDFs into text, and that step quietly loses tables, charts, layout and anything scanned badly. ColPali skips it. It looks at each page as an image and produces embeddings for small patches of that image, so a query about a figure in a chart can match the chart itself.

ColPali was introduced by Manuel Faysse and colleagues in ColPali: Efficient Document Retrieval with Vision Language Models, first posted to arXiv on 2024-06-27. The same paper introduced ViDoRe, a benchmark for retrieving visually rich pages across domains and languages.

How ColPali works

A vision language model reads the page image and outputs one vector per image patch. The query is embedded into one vector per token. At search time, each query token finds its closest patch vector on the page, and those maximum similarities are summed into the page score. This is the late-interaction scoring that ColBERT introduced for text, applied to images.

The authors report that ColPali largely outperforms modern text-extraction pipelines on ViDoRe while being simpler and end-to-end trainable. The training and inference code for the ColVision family, which also includes ColQwen2 and ColSmol variants, is MIT-licensed on GitHub.

The storage cost

One vector per patch means many vectors per page rather than one. Qdrant's 2026-10-13 post on Perplexity's pplx-embed-v2-late model counts about 1,700 vectors of 128 dimensions per page. It reports a 21x storage reduction with 1-bit quantization while keeping 96.5% of float32 retrieval quality without rescoring.

That cost is why multi-vector support became a 2026 database feature. Milvus 3.0 (2026-07-16) added a StructList type for ColBERT and ColPali vectors. Weaviate 1.40 (2026-10-14) added multi-vector search on its disk-based HFresh index through MUVERA, which encodes a set of vectors into one fixed-length vector for candidate search and then scores the finalists exactly with MaxSim.

Where ColPali fails, and what it is confused with

  • Clean text gains little

    For born-digital documents that extract cleanly, a text pipeline with hybrid search may match ColPali at a fraction of the storage.

  • Retrieval is only half

    ColPali returns pages. The model that answers still has to read those pages as images, which costs more tokens than text.

  • ColPali versus ColBERT

    ColBERT applies late interaction to text tokens. ColPali applies it to image patches, so it needs a vision model and handles layout that text never captured.

  • ColPali versus multimodal embeddings

    Models such as Gemini Embedding 2 put a whole image or page into a single vector in a shared space with text. That is cheaper to store but coarser than per-patch matching.

How Motiontic's engineers apply it

Motiontic's engineers reach for page-image retrieval when the eval set shows text extraction losing answers: scanned forms, slide decks, engineering drawings, financial tables. It is tested against a text pipeline on the same questions, and quantization or a reranking-only role is used to keep storage inside budget.

Three ways to retrieve from visually rich documents

Three ways to retrieve from visually rich documents
OCR then text RAGSingle-vector multimodalColPali-style multi-vector
What is embeddedExtracted text chunksWhole image or pageEvery image patch
Tables and chartsOften lost or garbledCaptured coarselyCaptured in detail
Vectors per pageA few chunksOneMany, one per patch
Storage costLowLowestHighest, cut by quantization
Needs OCRYesNoNo

Common questions

What is ColPali used for?

Finding the right page in large sets of PDFs, scans or slides where tables, charts and layout carry the answer and text extraction loses them.

Does ColPali need OCR?

No. ColPali embeds the page image directly, which is the point of the method. The answering model may still read extracted text alongside the page image.

What is the difference between ColPali and ColBERT?

Both use late interaction, matching each query token to its closest document vector. ColBERT works on text tokens, while ColPali works on image patches from a vision language model.

Which vector databases support ColPali embeddings?

Engines with multi-vector support, including Qdrant, Milvus 3.0 and Weaviate 1.40. Vespa also supports late interaction natively.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.