LLM bills tend to grow the same way cloud bills do. They creep up quietly, one feature at a time, until someone asks why the invoice tripled. Most of the waste sits in a few places, though, and the fixes are well understood. Below is the order we work through with clients, with pricing mechanics checked against official docs as of September 2026.
Step 1: Measure cost per feature, not per month
A monthly total tells you nothing actionable. Log these on every LLM call:
- Feature or endpoint name, and customer or tenant ID
- Model, input tokens, cached input tokens, output tokens (including reasoning tokens)
- Latency and whether the call was a retry
Then compute cost per feature and cost per successful outcome (per resolved ticket, per processed document). Just like with slow database queries, you will usually find that two or three call sites account for most of the spend.
A simple cost model is enough to start with.
cost_per_call = (
uncached_input * input_price
+ cached_input * input_price * cache_read_multiplier
+ cache_write_tokens * input_price * cache_write_multiplier
+ output_tokens * output_price
)Step 2: Cut tokens you did not mean to send
Before touching pricing tiers, get rid of the tokens you are paying for by accident.
Start with the system prompt. Old instructions pile up, and every token in it is paid for on every call.
Tool definitions are the next suspect. Each one is sent as input on every request, so an agent with 60 tools can spend thousands of tokens per call just describing them. Loading tools on demand fixes that. Anthropic reported one agent workflow dropping from about 150,000 to 2,000 tokens by loading tool definitions only when needed.
In long conversations, summarize or truncate the history instead of resending every turn.
Cap output too. Ask for structured, concise output and set a max token limit, because output tokens usually cost 4 to 5 times as much as input tokens.
Finally, watch the tokenizer. Model upgrades can change token counts for the same text, and Anthropic documents about 30% more tokens with the tokenizer on Claude 4.7 and later. Re-baseline costs whenever you switch models.
Step 3: Use prompt caching properly
Both OpenAI and Anthropic let you cache a prompt prefix so that repeated requests pay a fraction of the input price for the cached part. Reads cost roughly 0.1x the normal input rate. The mechanics differ between the two.
| OpenAI | Anthropic | |
|---|---|---|
| How it is enabled | Automatic | Automatic caching or explicit cache_control breakpoints |
| Cache write cost | Free on older models; 1.25x input on GPT-5.6 and later | 1.25x for the 5-minute cache, 2x for the 1-hour cache |
| Cache read cost | About 0.1x input (varies by model) | 0.1x input on most models, lower on some newest models |
| Lifetime | 30 minutes on GPT-5.6+ | 5 minutes by default, 1 hour optional |
In practice, the order of your prompt matters most. The system prompt goes first, followed by tool definitions, reference documents and few-shot examples. Put the user's message and anything else that varies at the end. The cache only matches an identical prefix.
Caching pays off with reuse. With a 1.25x write and 0.1x reads, a prefix written once and read once costs 1.35x its normal price instead of 2x. Read it ten times and you pay a fraction. A prefix used only once, on the other hand, costs more with caching.
Keep the prefix byte-identical as well. A timestamp or request ID near the top of the prompt silently disables caching.
Step 4: Move non-urgent work to batch
Both providers offer a batch API at 50% off input and output, with results returned asynchronously. OpenAI's window is 24 hours. Anthropic's batches usually finish in under an hour. Good candidates include:
- Nightly classification, tagging and enrichment
- Document and invoice processing that is not user-facing (see our invoice automation walkthrough)
- Running your eval suite
- Backfills and re-processing after a prompt change
Discounts stack. On Anthropic, batch pricing combines with cache read pricing. OpenAI also has a Flex tier billed at batch rates for synchronous requests that can tolerate slower responses and occasional capacity errors.
Step 5: Route to cheaper models behind evals
Many requests do not need your most capable model. Classification, extraction, short rewrites and routing decisions often work just as well on a small model that costs a fraction as much per token.
The catch is that you should never downgrade without an eval set. Build 100 to 300 real examples with expected outputs, score the cheap model against the expensive one, and only route the task types where quality holds. Re-run the evals on every prompt or model change. Skip this step and the money you save comes from shipping worse answers without knowing it. The same eval-first discipline applies to choosing between RAG and fine-tuning.
A common pattern is a cascade. You try the small model first, and escalate to the large one only when the small model's output fails validation or it reports low confidence. The router itself can be a cheap decision model. See how Jev handles model routing and other classification-style decisions.
Step 6: Cache answers, carefully
If users ask the same questions, cache the final answer. There are two ways to do it, with very different risks.
Exact-match caching maps a hash of the normalized input to a response. It is safe and simple.
Semantic caching returns a stored answer for a similar question. It can save more. It can also return confidently wrong answers for questions that look alike but differ in one important detail, so use it only with a high similarity threshold and on low-risk content.
Vendors quote big percentages for semantic caching. Measure the hit rate and error rate on your own traffic before trusting them.
Step 7: Set budgets and alerts
Give each tenant or user a daily token budget, so one runaway script or abusive user cannot burn a month's spend. Alert on cost per feature per day, in addition to the monthly total.
And set retry limits with backoff. Retry storms against an LLM API are expensive as well as slow.
Where to start this week
- Log tokens, cached tokens and cost per feature.
- Reorder prompts: static first, variable last. Confirm cache hits in the usage data.
- Move one non-urgent workload to the batch API.
- Build an eval set for your most expensive feature, then test a cheaper model on it.
- Add per-tenant budgets.
Prices and multipliers change often. Before you act on a specific number, check the official pricing pages, since the mechanics above are more durable than the prices themselves. Our cost calculator re-checks them weekly.