85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is AI red teaming?

AI red teaming is adversarial testing in which people, models or both deliberately attack an AI model or agent, through prompts, documents, tools and infrastructure, to find harmful behaviour and security weaknesses before real attackers or users do.

Red teaming comes from security and military practice: a team plays the adversary to find weaknesses the defenders missed. Anthropic's June 2024 post on the challenges of red teaming AI systems describes it as adversarially testing a system to identify potential vulnerabilities, and notes that the field lacks standard practices, which makes results from different organisations hard to compare.

For a chat model, red teaming mostly means prompts that draw out harmful output. For an agent, the attack surface widens to everything it reads and every tool it can call, so red teaming also covers prompt injection through documents and tool outputs, misuse of permissions and plain infrastructure flaws.

Kinds of AI red teaming

  • Domain-expert

    Policy experts probe usage-policy topics such as election integrity, and frontier-threat red teaming tests chemical, biological, radiological, nuclear, cyber and autonomy risks.

  • Multilingual and multicultural

    Testing in other languages and cultural contexts, where safeguards trained mostly on English can be weaker.

  • Automated

    One model generates attacks and another is trained to resist them. Open-source tools such as Microsoft's PyRIT and Promptfoo automate attack generation and scoring.

  • Multimodal

    Attacks that combine images and text.

  • Open-ended

    Crowdsourced testing and community events, such as the DEF CON AI Village challenges.

What Microsoft's AI red team found

Microsoft's AI red team, after more than 80 operations covering over 100 generative AI products by October 2024, published three lessons in January 2025. Generative AI amplifies existing security risks as well as adding new ones; one finding was an outdated FFmpeg component that enabled server-side request forgery, a conventional bug inside an AI product. People remain central, because automation helps with prompts, orchestration and scoring but cannot replace expertise and judgement. And defence in depth matters, through repeated break-fix cycles of red teaming, measurement and mitigation.

From red-team findings to evals

Anthropic describes a progression. Experts first probe a model by hand against a threat model and refine the inputs that work. A language model then generates hundreds or thousands of variations, turning a qualitative exercise into an automated, quantitative test that can run on every change, inside a loop of assessment, mitigation and retesting.

OWASP's GenAI Red Teaming Guide (January 2025) gives a structured, risk-based method that covers model-level issues such as toxicity and bias and system-level issues such as API misuse and data exposure, including agentic systems.

Limits to plan for

  • Never finished

    Microsoft describes AI red teaming as a continuous process that has to keep up with new risks.

  • Finds, does not measure

    A red team shows that an attack works, not how often it would succeed. Measurement comes from the evals built afterwards.

  • Only what is in scope

    An agent's tools, data sources and infrastructure have to be in scope, or the most likely attack paths go untested.

How Motiontic's engineers apply it

Motiontic's engineers red-team agents before launch against the client's own threat model: injection through each untrusted source, misuse of each high-risk tool and attempts to reach outside the sandbox. Attacks that work become regression tasks, so the same hole is checked on every later change, and findings go to the client's security team in the format their own reviews use.

Red teaming compared with evals

Red teaming compared with evals
Red teamingEvals
GoalFind failures no one has thought ofMeasure known behaviour repeatably
MindsetAdversarialSpecification
OutputFindings and working attacksScores against a baseline
CadenceBefore launch and periodicallyOn every change
WhoSecurity and domain experts, aided by attack toolsEngineers, with code, model and human graders

Common questions

What is AI red teaming?

AI red teaming is deliberately attacking an AI model or agent, by hand and with automated tools, to find harmful behaviour and security weaknesses before real attackers or users find them.

How is red teaming different from an eval?

Red teaming searches for new failures with an adversarial mindset. Evals measure known behaviour repeatably. Successful red-team attacks are usually turned into eval cases so they are retested on every change.

What tools are used for AI red teaming?

Open-source options include Microsoft's PyRIT and Promptfoo, which OpenAI announced it would acquire in March 2026 and which says it will remain open source. OWASP's GenAI Red Teaming Guide provides a method to go with them.

Can AI red teaming be fully automated?

Not according to Microsoft's AI red team, which found that automation scales the work but cannot replace human expertise, cultural knowledge and judgement.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.