85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is a semantic cache?

A semantic cache stores past language model responses keyed by the meaning of the prompt, and returns a stored answer when a new prompt's embedding is close enough to a cached one, skipping the model call.

An ordinary cache only hits when the request is identical. Users rarely ask identically: one asks how to return an item, another asks about the return policy. A semantic cache embeds each prompt and looks for a cached prompt that is close in meaning, then serves the stored response.

On a hit, there is no model call at all. That makes semantic caching different from prompt caching, which discounts a repeated prompt prefix but still runs the model every time.

How it works

  • Embed the prompt

    The incoming prompt becomes a vector with an embedding model.

  • Search past prompts

    A nearest-neighbour search runs over cached prompts, filtered by metadata such as tenant, locale and model version.

  • Apply a threshold

    If the closest match is within the distance threshold, the cached response is returned. If not, the model runs and the new prompt and response are written back.

  • Expire and evict

    Entries carry a time to live so stale answers age out, and cold entries are evicted under memory pressure.

The threshold problem

Redis's documentation names threshold tuning as the core difficulty: too loose and the cache serves wrong answers, too tight and the hit rate collapses. Two prompts can be close in embedding space and still need different answers, such as the same question about two different accounts.

Hard boundaries help. Scoping the cache by tenant, user role, locale and model version keeps reuse inside limits where a shared answer is safe.

Where it fits

  • Good fit

    High-volume, repetitive, non-personal questions: FAQ assistants, helpdesk deflection and internal policy lookups.

  • Poor fit

    Answers that depend on the user's own data, change by the hour, or trigger actions. An agent step that writes to a system should never be served from cache.

  • Options

    Redis LangCache is a managed semantic cache in preview on Redis Cloud, and RedisVL offers a self-managed SemanticCache. Any vector store with metadata filters and TTLs can host one.

How Motiontic's engineers apply it

Motiontic's engineers replay a sample of real traffic against a candidate threshold and have wrong hits graded before the cache goes live. The cache stays non-authoritative: it can be flushed at any time, and answers are invalidated when the source documents change.

Exact-match cache, prompt caching and semantic cache compared

Exact-match cache, prompt caching and semantic cache compared
Exact-match cachePrompt cachingSemantic cache
Hits whenThe request is identicalThe prompt starts with a cached prefixThe prompt means about the same
Model still runsNoYes, at a discountNo
Main riskLow hit rateCache misses when the prefix changesServing a wrong answer
Run byThe applicationThe model providerThe application

Common questions

What is the difference between semantic caching and prompt caching?

Prompt caching is a model-provider feature that reuses the processed prefix of a prompt, so the model still runs but costs less. A semantic cache sits in the application and returns a stored answer without calling the model when a similar prompt was seen before.

Can a semantic cache return wrong answers?

Yes, if the similarity threshold is too loose or the cache is shared across users whose answers differ. Metadata filters, conservative thresholds and graded replay tests reduce that risk.

Should AI agents use a semantic cache?

For repeated informational questions, it can cut cost and latency. For steps that act on systems or depend on live data, the agent should run fresh every time.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.