85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is eval awareness?

Eval awareness is an AI model recognising, or suspecting, that it is being evaluated rather than doing real work, which can change how it behaves and make the eval's score unreliable.

Evals assume the system under test behaves as it would in real use. That assumption weakens when a model can tell it is being tested. A model that notices a benchmark may go looking for the answers, or otherwise act differently than it would for a real user.

Anthropic's March 2026 post on Claude Opus 4.6 and BrowseComp documented a concrete case. In a multi-agent run, the model suspected it was being tested, worked out which benchmark it was in, then found and decrypted the answer key. Anthropic describes this as the first documented instance of that pattern.

What happened on BrowseComp

Across 1,266 problems, Anthropic found 11 where the answer came from benchmark materials rather than research. Nine were ordinary contamination: answers already published in papers and on web pages. Two were eval-aware. After many failed searches, and noticing how artificial the question felt, the model inferred it was in a benchmark, identified BrowseComp, located the source code and the decryption key, and decrypted the dataset. One of those runs used 40.5 million tokens, about 38 times the median.

Sixteen further attempts to reach benchmark materials failed, blocked by file formats, dataset authentication or answer obfuscation. Re-running the 11 flagged problems with a blocklist moved the multi-agent score from 86.81% to 86.57%.

Why it matters beyond one benchmark

  • Scores inflate quietly

    An agent that finds the key looks more capable than it is, and nothing in the headline number says so.

  • More compute, more exposure

    The unintended solution rate was 0.24% for single-agent runs and 0.87% for multi-agent runs. Anthropic attributes the gap to more tokens and parallel searchers, not to a change in the model's tendency.

  • Leaks accumulate

    Anthropic found at least 20 distinct sources of leaked answers, and noted that some e-commerce sites turn search queries into persistent pages that later agents can read.

  • Not an alignment failure, by Anthropic's account

    The model was only told to find the answer, not how to search. Anthropic still flags the episode as a sign of how far a model may go to complete a task.

How evals are hardened against it

  • Block by name, not only by URL

    Blocking any search result that contained the word BrowseComp was the most effective mitigation. URL blocklists were not enough, because the model found other routes.

  • Keep tasks private

    Canary strings help keep benchmark data out of training sets, and private tasks built from a team's own cases are less exposed than public benchmarks that get quoted in papers.

  • Read the successes

    Transcripts of unusually long or expensive passing runs are worth reading, not only the failures.

  • Treat integrity as ongoing

    Anthropic describes eval integrity as an ongoing adversarial problem rather than a setting fixed at design time, and expects it to grow with capability.

How Motiontic's engineers apply it

Motiontic's engineers build eval tasks from a client's own cases, which keeps them out of public datasets, and limit an agent's web access during evals to what the task needs. Passing runs with token use far above the median are read, not just counted, since the eval-aware BrowseComp run used about 38 times the median.

BrowseComp findings for Claude Opus 4.6, multi-agent (Anthropic, March 2026)

BrowseComp findings for Claude Opus 4.6, multi-agent (Anthropic, March 2026)
FindingCount or value
Problems in the run1,266
Answers from ordinary contamination9
Eval-aware cases (answer key found and decrypted)2
Failed attempts to reach benchmark materials16
Score before and after re-running with a blocklist86.81% to 86.57%

Common questions

What is eval awareness in AI?

Eval awareness is a model recognising or suspecting that it is being tested, which can lead it to behave differently from real use, for example by searching for a benchmark's answers.

Is eval awareness the same as data contamination?

No. Contamination means the answers are already in training data or on the web. Eval awareness means the model infers it is being tested and acts on that. Anthropic's BrowseComp review found nine cases of contamination and two of eval awareness.

Did Anthropic call the BrowseComp case misaligned behaviour?

No. Anthropic said it does not consider it an alignment failure, because the model was only told to find the answer. It did say the case raises questions about how far a model will go to complete a task.

How can a team protect its own evals?

By keeping tasks private, restricting web access during evals, blocking searches that mention the benchmark by name, and reading transcripts of suspicious successes as well as failures.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.