Every AI operations budget conversation starts with a price list and ends with a surprise. The per-token prices on that list have been falling for three straight years, yet the invoice keeps growing — because the cost of an AI agent was never the price of a token. It is how many tokens you make the model read to reach one decision. Teams that model spend as price × users are off by an order of magnitude. Teams that model it as tokens per resolved incident × incidents usually cut the bill in half within a quarter without degrading a single answer.
The unit price is a distraction; the token volume is the bill
A triage agent does not make one call. It makes a conversation.
Take a mid-size shop: 18,000 alerts a month reach an agent that reads a 30,000-token log slice alongside an 8,000-token system prompt and tool schema. That first turn is 38,000 input tokens. If the agent takes ten turns — pull metrics, check deploys, correlate, draft, verify — and each turn re-sends the growing history, the run lands around 560,000 input tokens for a single incident. At $3 per million input tokens that is $1.68. Multiply by 18,000 incidents and you are at roughly $30,000 a month for triage alone, before a single remediation executes. Nobody approves $30,000 of triage by reading a price list that says "$3 per million."
Context re-sending is the silent multiplier
Of those 560,000 tokens, roughly 80,000 are the identical system prompt and tool schema, sent ten times unchanged. Another large block is the same log slice quoted back on every turn. The model has no memory between calls; the harness rebuilds the world each time and pays for it each time.
Prompt caching is the cheapest fix in the stack. Cache the static prefix and most providers bill those tokens at a fraction — often a tenth — of the normal input rate. That turns 80,000 full-price tokens into 8,000 effective ones and shaves roughly 13% off the run before you touch anything else. Then prune: summarise turns one through six into 800 tokens instead of carrying them verbatim, and drop tool results the agent has already acted on.
Fan-out and retries are the costs nobody forecasts
Two multipliers hide behind the baseline. The first is fan-out: a correlation step that queries twelve services in parallel, each returning 20,000 tokens of metrics, adds 240,000 input tokens to the run — a 43% increase — and most of it is noise the agent never cites. The second is retries: if 15% of tool calls time out and get replayed with full context, you pay for the same tokens twice. Together, fan-out and retries routinely add 60–90% to the number you modelled.
A bill you cannot attribute to one incident, one playbook, and one team is a bill you can only argue about, not manage.
Route by difficulty, not by habit
Most operational reasoning is not hard. In a mature deployment, roughly 70% of agent calls are classification, deduplication, summarisation, or a lookup against a known runbook — work a small model does as well as a frontier one. Tier the calls:
| Tier | Handles | Typical cost per call |
|---|---|---|
| Deterministic rules | known signatures, restarts, ack/close | $0.00 |
| Small model | classification, dedupe, summarisation | ~$0.0005 |
| Frontier model | novel correlation, root-cause narrative, remediation plan | ~$0.02 |
Run the arithmetic on 1,000 agent calls a day at a 70/25/5 split: 250 small-model calls at $0.0005 plus 50 frontier calls at $0.02 comes to about $1.13 a day. Send all 1,000 to the frontier model and you spend $20. Same answers on the easy 70%, roughly 18x cheaper.
Put the ceiling inside the loop, not in the dashboard
Finance tooling tells you what you spent last month. That is a post-mortem, not a control. The budget has to live in the agent runtime, enforced per run:
- Max turns per incident. Twelve is usually enough; a run that hits the cap should escalate to a human, not loop.
- Token ceiling per incident. Derive it from the p95 of your own successful runs, not from a vendor default.
- Deterministic-first routing. Anything with a known signature — disk pressure, a flapping health check — never reaches a model at all.
- Attribution tags on every call. Incident ID, team, playbook, model tier. Without these, the rest is guesswork.
- Alert on per-incident cost outliers, not just aggregate spend. One runaway loop is easier to find than a 4% drift across a thousand healthy runs.
Attribution changes behaviour faster than policy does. Teams that see their own per-incident cost start pruning their own prompts.
Where this is heading: cost as a runtime signal
The next step is not a cheaper model — it is an agent that knows what its own reasoning costs. Budget-aware agents will check remaining headroom before a fan-out, downgrade to a small model when confidence is already high, and abandon an investigation that is burning tokens without shrinking uncertainty. Cost stops being a finance metric and becomes an SLO alongside latency and success rate. "Resolve 95% of incidents under $2.50" is a target you can engineer against, and a far better design constraint than an invoice you read after the fact.