What is a semantic cache?
A semantic cache stores past language model responses keyed by the meaning of the prompt, and returns a stored answer when a new prompt's embedding is close enough to a cached one, skipping the model call.
An ordinary cache only hits when the request is identical. Users rarely ask identically: one asks how to return an item, another asks about the return policy. A semantic cache embeds each prompt and looks for a cached prompt that is close in meaning, then serves the stored response.
On a hit, there is no model call at all. That makes semantic caching different from prompt caching, which discounts a repeated prompt prefix but still runs the model every time.
How it works
Embed the prompt
The incoming prompt becomes a vector with an embedding model.
Search past prompts
A nearest-neighbour search runs over cached prompts, filtered by metadata such as tenant, locale and model version.
Apply a threshold
If the closest match is within the distance threshold, the cached response is returned. If not, the model runs and the new prompt and response are written back.
Expire and evict
Entries carry a time to live so stale answers age out, and cold entries are evicted under memory pressure.
The threshold problem
Redis's documentation names threshold tuning as the core difficulty: too loose and the cache serves wrong answers, too tight and the hit rate collapses. Two prompts can be close in embedding space and still need different answers, such as the same question about two different accounts.
Hard boundaries help. Scoping the cache by tenant, user role, locale and model version keeps reuse inside limits where a shared answer is safe.
Where it fits
Good fit
High-volume, repetitive, non-personal questions: FAQ assistants, helpdesk deflection and internal policy lookups.
Poor fit
Answers that depend on the user's own data, change by the hour, or trigger actions. An agent step that writes to a system should never be served from cache.
Options
Redis LangCache is a managed semantic cache in preview on Redis Cloud, and RedisVL offers a self-managed SemanticCache. Any vector store with metadata filters and TTLs can host one.
How Motiontic's engineers apply it
Motiontic's engineers replay a sample of real traffic against a candidate threshold and have wrong hits graded before the cache goes live. The cache stays non-authoritative: it can be flushed at any time, and answers are invalidated when the source documents change.
Exact-match cache, prompt caching and semantic cache compared
| Exact-match cache | Prompt caching | Semantic cache | |
|---|---|---|---|
| Hits when | The request is identical | The prompt starts with a cached prefix | The prompt means about the same |
| Model still runs | No | Yes, at a discount | No |
| Main risk | Low hit rate | Cache misses when the prefix changes | Serving a wrong answer |
| Run by | The application | The model provider | The application |
Common questions
What is the difference between semantic caching and prompt caching?
Prompt caching is a model-provider feature that reuses the processed prefix of a prompt, so the model still runs but costs less. A semantic cache sits in the application and returns a stored answer without calling the model when a similar prompt was seen before.
Can a semantic cache return wrong answers?
Yes, if the similarity threshold is too loose or the cache is shared across users whose answers differ. Metadata filters, conservative thresholds and graded replay tests reduce that risk.
Should AI agents use a semantic cache?
For repeated informational questions, it can cut cost and latency. For steps that act on systems or depend on live data, the agent should run fresh every time.
Related
Prompt caching
Paying less for the part of the prompt that does not change between calls.
Embeddings
Meaning turned into vectors, so similar things can be found by distance.
Vector database
Storage and nearest-neighbour search for embeddings, now in many shapes.
Customer service agents
Support agents that look up, act and escalate inside Zendesk, Intercom and the systems behind them.
AI product development
New products and major AI features, built from scratch alongside the client's team.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.