The first time a monthly inference bill crosses into four figures, the reflex is almost always the same: reach for a smaller model. Half the price per token, ship it, done. It's also usually the wrong first move. Downgrading the model trades quality you can measure for savings you can only estimate, and it does nothing about the structural waste that inflated the bill in the first place.
The larger, safer wins come before you touch the model at all. They're mechanical, they compound, and stacked together they routinely take a bill down by something close to half. None of them require a fine-tune, a new vendor, or a quality regression you have to defend in a review.
Here's the arithmetic, run against a hypothetical $10,000/month bill on a retrieval-augmented chat product — roughly the shape a lot of teams have right before they scale.
Read the bill by shape, not by total
You cannot cut a number you can't decompose. Before anything else, tag every LLM call with three fields: input tokens, output tokens, and a call-site label (chat_turn, nightly_summarize, eval_run, classify). Within a day you'll have a table that answers the only question that matters: which shapes of traffic are actually spending the money.
Almost every bill I've audited splits along the same fault lines. A big chunk of input tokens is a stable prefix — the system prompt, few-shot examples, and retrieved documents that repeat every turn. A meaningful fraction of calls aren't waiting on a human. And a surprising share of the total is output tokens, which most providers price at three to five times the input rate. Each of those is a lever.
Lever 1 — Cache the prefix that never changes
In a RAG or agent workload, input dwarfs output, and most of that input is identical request to request. Prompt caching bills that repeated prefix at a steep discount — cache reads run roughly a tenth of the base input price on Anthropic, and OpenAI discounts cached input automatically once a prompt is long enough.
The catch is that caching is a prefix match: one byte changing anywhere in the prefix invalidates everything after it. So the entire game is ordering. Stable content first, volatile content last.
# Stable prefix first, marked cacheable; volatile question last, unmarked.
system = [{
"type": "text",
"text": SYSTEM_PROMPT + RETRIEVED_DOCS, # identical across the session
"cache_control": {"type": "ephemeral"},
}]
messages = [{"role": "user", "content": user_question}] # changes every turn
The classic own-goal is interpolating datetime.now() or a request ID into the system prompt — the prefix is then unique every call and the cache never hits. Verify with the cache_read_input_tokens field in the response; if it's zero across identical prefixes, something upstream is mutating the bytes. On a workload where 70% of input is a repeated prefix, this alone can knock 30–40% off the input line. Call it $1,800/month. Running total: $8,200.
Lever 2 — Batch everything that isn't waiting on a human
Both major providers offer a Batch API: submit requests as a job, accept results within 24 hours (often within one), pay 50% less on every input and output token. It is the single highest-return line change in this list, and it costs you nothing but latency you weren't using.
The mistake is assuming batch is only for offline ETL. Look again at your call-site labels. Nightly summarization, backfills, evaluation runs, content classification, embedding refreshes — none of it is watching a spinner. If 40% of your spend is non-interactive and you move all of it to batch, that 40% halves. On the running $8,200, non-interactive traffic worth about $3,280 becomes ~$1,640. Savings: $1,640. Running total: $6,560.
Lever 3 — Route by difficulty, not by default
Now you get to think about model size — but as a router, not a global downgrade. Most traffic is easy: a short classification, a formatting pass, a "did the user say yes." A small model handles those at a fraction of the price, and the gap between tiers is large — often five to thirty times per token. The trick is sending the hard queries to the capable model and only those.
A cheap, effective router is a length-and-keyword heuristic with a confidence escape hatch: run the small model first, and escalate to the frontier model only when its output is low-confidence or the task trips a complexity flag. If 60% of queries route to a model that's an order of magnitude cheaper, the blended per-query cost on that slice collapses. Conservatively, another $1,300 off the interactive remainder. Running total: $5,260.
Lever 4 — Pay for the output you actually need
Output tokens are the expensive ones. Two habits waste them: an unbounded max_tokens, and prompts that invite the model to narrate. Cap the ceiling to what a correct answer needs, and instruct for terseness — "answer in one sentence, no preamble." For structured extraction, use the provider's JSON/schema mode so you're paying for fields, not for a chatty wrapper around them. Trimming average output length 25% on a bill where output is a third of the total is another ~$400. Running total: $4,860 — a 51% cut, with the model quality on your hard path untouched.
Where each lever bites, and where it doesn't
The arithmetic above is illustrative, not a promise — your mix determines which levers pay. Caching does nothing for one-shot prompts with no shared prefix, and the longer-lived cache tier costs more to write, so it only pays under sustained traffic. Batch is useless for anything a user is waiting on. Routing adds a failure mode — a mis-routed hard query gets a worse answer — so gate it behind a confidence check and watch the escalation rate. And output caps truncate mid-thought if set too low; measure stop_reason before tightening them.
The through-line: these are boring, reversible, quality-neutral changes, and that is exactly why they should come first. A smaller model is a bet on your eval coverage. Caching, batching, routing, and output discipline are just refusing to pay full price for tokens you were already going to spend.
The concrete takeaway: don't touch your model this quarter. Instrument every call with input tokens, output tokens, and a call-site label, let it run for a week, then apply the four levers in the order your own table ranks them by spend. You will almost certainly find your half before you have to argue about quality — and you'll have the instrumentation to prove it when you finally do change models.
Sources: Anthropic — Prompt caching, Anthropic — Message Batches API, OpenAI — Batch API, OpenAI — Prompt caching