What is the agent quality flywheel?
The agent quality flywheel is a continuous loop in which an AI agent is evaluated, shipped, monitored in production and fixed, with each production failure turned into a test case for the next cycle.
One-off testing assumes an agent is finished when it ships. Agents are not: users ask new things, tools change and models are upgraded. The flywheel framing, which Google uses for its Gemini Enterprise Agent Platform evaluation tooling, treats quality as a loop rather than a gate at the end.
Google's June 2026 developer blog post describes three phases introduced at Cloud Next '26: Build and Test, Ship and Monitor, and Learn and Refine. Each turn of the loop makes the next one cheaper, because every failure found in production arrives as a ready test case.
The three phases
Build and Test
Prepare eval data, run the agent, grade the results, analyse failures, apply a targeted fix and re-run against the previous baseline.
Ship and Monitor
Online monitors grade live traffic and write quality scores into the monitoring system, so drift shows up without waiting for complaints.
Learn and Refine
Failing production traces go back into Build and Test as new cases, and the cycle repeats.
Build and Test in five stages
Google's eval skill expands the first phase into five stages. Prepare data, from OpenTelemetry traces, hand-written cases or synthesised scenarios. Run inference to produce traces, a step skipped when production traces already exist. Grade, with model-based AutoRaters or custom metrics; this is the only stage that always runs. Analyse failures by reading rubric verdicts, clustering them automatically once there are ten or more. Then optimise and iterate: apply a targeted fix, rerun and compare.
In Google's travel-concierge example, one instruction change moved a revision-handling metric from 21% failures to 5%.
Where the flywheel stalls
Model scores treated as truth
Google says AutoRater scores are directional signals and the deltas between runs matter more than absolute values.
Synthetic data only
Generated scenarios bootstrap the loop; Google notes production data is what makes it useful.
Fixes with no human approval
Google's skill proposes changes and a person approves them. An optimiser left alone can overfit the suite it is graded on.
No regression gate
Without a regression suite, a fix for one failure can quietly break a case that was already solved.
Same idea, other names
Anthropic's January 2026 eval guide describes the same compounding effect without the flywheel label: failures become test cases, capability tasks graduate into regression suites, and automated evals are layered with production monitoring, A/B tests, user feedback and transcript review. It compares the layers to Swiss cheese: no single layer catches everything.
How Motiontic's engineers apply it
Motiontic's engineers set up the loop before launch: production traces in a format the eval harness can read, a triage step that turns confirmed failures into tasks, and a regression gate in CI. The client's team owns the loop after handover, so quality keeps compounding after the engagement ends.
Where each phase gets its signal
| Phase | Input | Output |
|---|---|---|
| Build and Test | Eval dataset or existing traces | Graded baseline and targeted fixes |
| Ship and Monitor | Live traffic | Online quality scores and failing traces |
| Learn and Refine | Failing production traces | New eval cases for the next cycle |
Common questions
What is an agent quality flywheel?
It is a continuous loop of testing, shipping, monitoring and fixing an AI agent, in which each production failure becomes a test case so the next cycle starts from a stronger suite.
How is the flywheel different from a regression suite?
The regression suite is one part of it: the set of solved cases that must keep passing. The flywheel is the loop that keeps feeding new cases from production into the suites and checks every fix against them.
Does the flywheel require Google's platform?
No. Google's tooling implements it, but the loop needs only traces, an eval harness, graders and a routine for turning failures into tasks, all of which open-source and commercial tools support.
Can the loop be fully automated?
Grading, clustering and re-running can be. Google's own skill still has a person approve each proposed change, and treats model-based scores as directional rather than final.
Related
Online evals
Grading live traffic in production, not only test sets before release.
Regression eval
The suite that catches what a change broke before users do.
Agent observability
Step-by-step traces of what an agent did, why, and what it cost.
AgentOps
DevOps and MLOps, adapted for systems that decide their own next step.
AI agent development
Agents that do real work for users, with the evals, traces and guardrails to keep them in production.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.