What are online evals, and how do they differ from offline evals?
Online evals score an AI system's live production traffic with automated graders as it runs, while offline evals score a curated dataset in development or CI before a change ships.
Offline evals answer whether a change is safe to ship, using cases a team chose in advance. Production does not stay inside those cases. Users phrase things differently, upstream data changes, a provider updates a model, and new failure types appear that no test set anticipated.
Online evals run the same kind of graders on live traces, usually on a sample, so a drop in quality shows up as a metric instead of a complaint.
Offline and online, side by side
Offline evals run a golden dataset with reference answers in development and CI, and gate releases. Online evals run on real traffic after release, usually without a reference answer, so they rely on reference-free checks: rubric-based LLM judges, policy and format checks, tool error rates and user signals.
Vendors now ship both in one place. Pydantic Logfire launched online evals on 2026-04-30, scoring live traffic with the same evaluators as offline Pydantic Evals and emitting the OpenTelemetry gen_ai.evaluation.result event. Anthropic's eval guide lists LangSmith for offline and online evals, and LangSmith Engine (beta) clusters production failures and generates evaluators from them.
What online evals catch
Input drift
New topics, languages or request types that the test set never covered.
Upstream changes
A data source, API or retrieved document that changed shape or content.
Model changes
Behaviour shifts after a provider update or a model swap.
Slow regressions
A gradual fall in a quality score that no single release caused.
Practical constraints
Sampling
Grading every trace with an LLM judge is expensive, so most teams grade a sample and every flagged trace.
Off the request path
Graders run asynchronously so they add no latency for the user.
Privacy
Production traces carry personal data, so redaction and access rules apply to the grading pipeline too.
How Motiontic's engineers apply it
Motiontic's engineers reuse offline graders online wherever they work without a reference answer, so a score means the same thing before and after release. Low-scoring production traces are reviewed weekly, and the confirmed failures are added to the golden dataset and the regression suite.
Offline evals compared with online evals
| Offline evals | Online evals | |
|---|---|---|
| Data | Curated dataset with references | Sampled live production traces |
| When | In development and CI, before release | Continuously, after release |
| Graders | Code checks, reference matching, LLM judges | Reference-free judges, policy checks, user signals |
| Answers | Is this change safe to ship? | Is quality holding up in real use? |
| Main risk | Test set drifts from real usage | Cost, privacy and no ground truth |
Common questions
What are online evals?
Online evals are automated quality checks run on an AI system's live production traffic, usually on a sample of traces, to track whether quality holds up after release.
What is the difference between online and offline evals?
Offline evals score a fixed, curated dataset before a change ships. Online evals score real traffic after it ships. Offline evals gate releases; online evals catch drift and new failures.
Do online evals replace offline evals?
No. They answer different questions. Anthropic's guidance treats automated pre-launch evals and production monitoring as separate layers, and notes that no single layer catches everything.
Related
Eval
Repeatable, scored tests that tell a team whether an AI change made things better or worse.
Agent observability
Step-by-step traces of what an agent did, why, and what it cost.
Regression eval
The suite that catches what a change broke before users do.
Golden dataset
The reviewed, versioned test cases an eval trusts as ground truth.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.