85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What are online evals, and how do they differ from offline evals?

Online evals score an AI system's live production traffic with automated graders as it runs, while offline evals score a curated dataset in development or CI before a change ships.

Offline evals answer whether a change is safe to ship, using cases a team chose in advance. Production does not stay inside those cases. Users phrase things differently, upstream data changes, a provider updates a model, and new failure types appear that no test set anticipated.

Online evals run the same kind of graders on live traces, usually on a sample, so a drop in quality shows up as a metric instead of a complaint.

Offline and online, side by side

Offline evals run a golden dataset with reference answers in development and CI, and gate releases. Online evals run on real traffic after release, usually without a reference answer, so they rely on reference-free checks: rubric-based LLM judges, policy and format checks, tool error rates and user signals.

Vendors now ship both in one place. Pydantic Logfire launched online evals on 2026-04-30, scoring live traffic with the same evaluators as offline Pydantic Evals and emitting the OpenTelemetry gen_ai.evaluation.result event. Anthropic's eval guide lists LangSmith for offline and online evals, and LangSmith Engine (beta) clusters production failures and generates evaluators from them.

What online evals catch

  • Input drift

    New topics, languages or request types that the test set never covered.

  • Upstream changes

    A data source, API or retrieved document that changed shape or content.

  • Model changes

    Behaviour shifts after a provider update or a model swap.

  • Slow regressions

    A gradual fall in a quality score that no single release caused.

Practical constraints

  • Sampling

    Grading every trace with an LLM judge is expensive, so most teams grade a sample and every flagged trace.

  • Off the request path

    Graders run asynchronously so they add no latency for the user.

  • Privacy

    Production traces carry personal data, so redaction and access rules apply to the grading pipeline too.

How Motiontic's engineers apply it

Motiontic's engineers reuse offline graders online wherever they work without a reference answer, so a score means the same thing before and after release. Low-scoring production traces are reviewed weekly, and the confirmed failures are added to the golden dataset and the regression suite.

Offline evals compared with online evals

Offline evals compared with online evals
Offline evalsOnline evals
DataCurated dataset with referencesSampled live production traces
WhenIn development and CI, before releaseContinuously, after release
GradersCode checks, reference matching, LLM judgesReference-free judges, policy checks, user signals
AnswersIs this change safe to ship?Is quality holding up in real use?
Main riskTest set drifts from real usageCost, privacy and no ground truth

Common questions

What are online evals?

Online evals are automated quality checks run on an AI system's live production traffic, usually on a sample of traces, to track whether quality holds up after release.

What is the difference between online and offline evals?

Offline evals score a fixed, curated dataset before a change ships. Online evals score real traffic after it ships. Offline evals gate releases; online evals catch drift and new failures.

Do online evals replace offline evals?

No. They answer different questions. Anthropic's guidance treats automated pre-launch evals and production monitoring as separate layers, and notes that no single layer catches everything.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.