What is temperature in an LLM?
Temperature is a sampling parameter that sets how a language model picks each next token: low values favour the most likely tokens for steadier output, and higher values spread the choice for more varied output.
At each step a language model produces a probability for every possible next token. Temperature rescales those probabilities before one is drawn. Near zero, the most likely token almost always wins. Above one, less likely tokens get picked more often. Two related settings narrow the pool: top_p keeps the smallest set of tokens whose probabilities add up to a threshold, and top_k keeps a fixed number of the most likely ones.
For years, setting temperature to zero was the standard advice for extraction and classification. In 2026 that advice ran out. The three largest model vendors have removed or deprecated these parameters on their newest models, and requests that still set them can fail.
What changed in 2026
Google
The Gemini API changelog for 21 July 2026 marks temperature, top_p and top_k as deprecated.
OpenAI
The API changelog entry for GPT-6 Astra (3 September 2026) states that the model does not support custom temperature or top_p values, or log probabilities.
Anthropic
API version 2026-10-21 removes temperature, top_p and top_k; requests that set them return a 400 error. On earlier API versions, Claude 4.7 and later models already rejected non-default values. Anthropic's advice is to omit them and guide behaviour through prompting.
What replaces it
Structured outputs for format
Where temperature zero was used to keep JSON clean, a schema enforced by constrained decoding is the stronger guarantee.
Prompts and examples for tone
Variety or restraint in wording is now asked for directly, with examples of the wanted output.
Effort for depth
On reasoning models, the effort setting is the main dial on how much the model works on a request.
Evals for variance
Output still varies between runs. Running each eval case several times and tracking pass rates shows how consistent a system really is.
Migration risks
The breakage is usually indirect. Agent frameworks, SDK wrappers and observability proxies often set a default temperature on every request. When the API stops accepting the parameter, calls fail even though no one on the team wrote it. Searching the codebase and its configuration for sampling parameters is the first step before an SDK or model upgrade.
Inference servers for open-weight models typically still expose temperature and its relatives, so teams that self-host keep the knob, along with the responsibility for choosing its value.
How Motiontic's engineers apply it
On client systems, Motiontic's engineers strip sampling parameters from calls to current hosted models, replace format control with structured outputs, and measure run-to-run variance with repeated eval trials before and after the change.
Sampling parameters at the three largest vendors (as of November 2026)
| Vendor | Status | Date | Suggested replacement |
|---|---|---|---|
| Google (Gemini API) | temperature, top_p, top_k deprecated | 2026-07-21 | Model defaults and prompting |
| OpenAI (GPT-6 Astra) | No custom temperature or top_p | 2026-09-03 | Effort levels and prompting |
| Anthropic (Claude API) | Removed in API version 2026-10-21; 400 error if set | 2026-10-21 | Omit and use prompting |
Common questions
What does temperature do in an LLM?
It controls how randomly the model picks each next token. Low temperature makes the most likely token win almost every time; high temperature lets less likely tokens through, which gives more varied text.
What temperature should be used for an AI agent?
On current hosted frontier models, none: Anthropic, OpenAI and Google have removed or deprecated custom temperature on their newest models. Consistency now comes from schemas, prompts, effort settings and evals.
Does temperature zero make output deterministic?
It makes output more repeatable, not guaranteed identical. Systems that need fixed formats should enforce them with structured outputs and validation rather than rely on sampling settings.
Why are vendors removing temperature?
The vendors' notes do not give a full rationale. Anthropic's deprecation table says to omit the parameters and guide behaviour through prompting, and the newest models are steered through effort settings and output schemas instead.
Related
Structured outputs
JSON that matches the schema because the decoder allows nothing else.
Reasoning model
A model that thinks before it answers, at a price in tokens and time.
pass@k and pass^k
Can the agent do it at all, versus does it do it every time.
Eval
Repeatable, scored tests that tell a team whether an AI change made things better or worse.
AI product development
New products and major AI features, built from scratch alongside the client's team.
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.