What is multimodal RAG?
Multimodal RAG is retrieval-augmented generation over content that is not only text, such as page images, charts, scans, audio or video, where both retrieval and the answering model handle those formats.
Checked
Much of what businesses know lives in formats a text pipeline flattens: scanned contracts, slide decks, engineering drawings, recorded calls, inspection photos. Converting all of it to text loses tables, layout and anything the converter misreads.
Multimodal RAG keeps more of the original. Since 2024 the pieces have matured on both sides: embedding models that place images, audio and video in the same space as text, retrievers that read page images directly, and answering models that read PDFs page by page as images.
Three architectures
Convert to text
OCR, captions and transcripts feed an ordinary text RAG pipeline. Cheapest to store and easiest to search with BM25, but tables, charts and layout are often lost.
Shared multimodal embeddings
One model embeds every format into one space. Google's Gemini Embedding 2 (2026-03-10) maps text, images, video, audio and PDFs into a single space, with limits of 6 images per request, 120 seconds of video and 6 PDF pages per input.
Page-image late interaction
ColPali-style models embed each page image as many patch vectors. Qdrant's 2026-10-13 post counts about 1,700 vectors per page for Perplexity's pplx-embed-v2-late, which is why quantization matters here.
The answering side
Retrieval only finds the page or clip. The model then has to read it. Claude's PDF support, for example, converts each page into an image and supplies the extracted text alongside it. Anthropic's docs put text at roughly 1,500 to 3,000 tokens per page, plus image tokens for each page, with up to 600 pages per request, or 100 when the context window is under 1M tokens.
That cost shapes the design: retrieve precisely, then send only the pages that matter.
Where multimodal RAG fails, and what it is confused with
Token cost
Image pages cost far more to read than text chunks. Sending ten full pages where two paragraphs would do inflates cost and latency.
Chunking audio and video
Clips need boundaries too. Cut in the wrong place and a question about a moment in a call misses it, just as text chunking can split an answer.
Citations
Pointing to a page is easy; pointing to the right cell of a table or second of a video is harder, and reviewers need that precision.
Versus a multimodal model
A model that can read images is not multimodal RAG on its own. The retrieval layer also has to index and find non-text content.
How Motiontic's engineers apply it
Motiontic's engineers start by checking where a text pipeline actually loses answers on the client's documents. Only those document types move to image-based retrieval, and the answering model receives the few pages that retrieval ranks highest, which keeps per-question cost predictable.
Multimodal RAG architectures compared
| Convert to text | Shared multimodal embedding | Page-image multi-vector | |
|---|---|---|---|
| What is indexed | OCR text, captions, transcripts | One vector per image, clip or page | Many vectors per page |
| Keeps layout and charts | Poorly | Coarsely | In detail |
| Keyword search | Yes | No, unless text is also indexed | No, unless text is also indexed |
| Storage | Lowest | Low | Highest |
| Example | Any text RAG stack | Gemini Embedding 2 | ColPali, pplx-embed-v2-late |
Common questions
What is multimodal RAG?
Retrieval-augmented generation where the knowledge base includes images, scans, charts, audio or video, and both the retriever and the answering model can work with those formats.
Do PDFs need OCR for RAG?
Not necessarily. Page-image retrievers such as ColPali search the page images directly, and models such as Claude read PDF pages as images plus extracted text.
Which embedding models handle images, audio and video?
Google's Gemini Embedding 2, released in preview on 2026-03-10, maps text, images, video, audio and PDFs into one embedding space. Page-image models such as ColPali focus on documents.
Is multimodal RAG more expensive than text RAG?
Usually. Image pages cost more tokens to read and multi-vector indexes cost more to store, so precise retrieval and quantization matter more than in text-only systems.
Related
ColPali
Searching PDFs as images, with one vector per patch instead of extracted text.
Late interaction (ColBERT)
Many vectors per passage, matched token by token: finer retrieval, more storage.
Chunking (RAG)
Splitting documents into passages before indexing, which quietly sets retrieval quality.
Retrieval-augmented generation (RAG)
Fetch the right passages at query time, then answer from them with citations.
AI product development
New products and major AI features, built from scratch alongside the client's team.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.