FinOps for AI spend: tokens, caching, routing and budgets that hold

AI bills behave differently from cloud bills. There is no idle server to switch off at night; every question a customer asks, every document an agent reads and every retry in a loop is paid by the token, and one feature with a long system prompt can quietly cost more than the rest of the platform. The good news: the same discipline that cuts cloud costs works here, with a few new levers. This is the order I use, with the AI guide on this site as a running example.

Dominik Heinrich

November 8, 202611 min read

1. See the cost per feature, not per provider

The invoice from an AI provider tells you how much you spent, not on what. Every API response includes token counts for input and output, and that is all you need: log them with the feature that made the call, the team that owns it and, in a multi-tenant product, the customer. The FinOps Foundation's work on FinOps for AI starts in the same place as classic FinOps: allocation before optimization.

Once the log exists, one daily report per feature changes the conversation. "AI costs went up" becomes "the document summary costs four times what the chat costs, and most of it is one customer re-running the same export". That is a sentence a product owner can act on.

javascript

// One log line per AI call: enough to report cost per feature every day
const res = await callModel(request);
log.info({
  event: "ai_usage",
  feature: "doc-summary",      // which product feature made the call
  team: "documents",
  tenant: request.tenantId,
  model: res.model,
  input_tokens: res.usage.input_tokens,
  output_tokens: res.usage.output_tokens,
  cached_tokens: res.usage.cached_tokens ?? 0,
});

How AI spend gets controlled: product features send tagged requests to an AI gateway with token limits, budgets, prompt and answer caches and a router; the router sends easy requests to a small model, hard ones to a large model, non-urgent work to batch APIs and steady volume to a self-hosted model; usage flows into a cost dashboard per feature with budget alerts and a daily report to the team chat. — AI spend under control: tag every request, cache and route in a gateway, batch what can wait, report per feature every day.

2. Know what you pay for in a token bill

Four things drive the bill. Input tokens: everything you send, including the system prompt, retrieved documents and the full conversation history, again on every turn. Output tokens: usually priced well above input. Reasoning or "thinking" tokens: on models that think before they answer, those are billed too; Google's Gemini price list states the output price including thinking tokens. And cached input, which is much cheaper than fresh input when the provider can reuse it.

The biggest silent cost is usually history. A chat that resends twenty earlier turns pays for them twenty times. The guide on this site keeps the last twelve turns, cuts single messages at 1,500 characters and caps the answer length; on Gemini it also sets a low thinking level, which is fast and enough for a website guide.

Tip from practice

Set a maximum output length per feature. A chat answer and a contract summary need very different limits, and an unbounded output is how one bad prompt turns into an expensive afternoon.

3. Cache what repeats

Most AI traffic repeats more than it seems. Provider-side prompt caching rewards a stable prefix: put the long, unchanging part first, the knowledge base or instructions, and the per-request part after it. Anthropic bills cache reads at a fraction of the normal input price, a tenth on most models; OpenAI applies caching automatically and discounts reused input tokens by up to 95 percent; Gemini has implicit caching. None of that works if a timestamp or a random ID sits at the top of the prompt.

The second cache is your own. Many visitors ask the same first question, so the guide on this site keeps first-turn answers for six hours and serves its suggested starter questions from pre-written answers that need no API call at all. Those answers appear instantly and cost nothing per visit.

4. Route: the smallest model that does the job

Not every request needs the best model. Classifying a ticket, extracting a date or answering a question from a short FAQ works with a small, fast model at a fraction of the price; reasoning over a long contract or planning a multi-step agent task may need the large one. A router in front of the providers decides per request, by feature, by length or by a quick classification, and escalates only when the small model isn't good enough.

Decide with data, not with a hunch: keep a small set of real questions with the answers you expect, run it against the candidate models and compare quality per euro. The guide on this site runs on a fast model with an even lighter model as fallback; when the free tier's daily quota for the main model runs out, the lighter one keeps answering instead of the chat going silent.

5. Batch everything that can wait

Nightly document processing, re-tagging a catalogue, evaluating last week's support answers: none of that needs an answer in two seconds. Batch APIs take a file of requests and return the results within a day, at a discount. OpenAI's Batch API advertises 50 percent lower costs with a 24-hour turnaround, Google's Gemini Batch API processes requests at 50 percent of the standard cost with a 24-hour target, and Anthropic offers a discounted Batch API as well.

The pattern is simple: anything a user isn't waiting for goes into a queue, the queue is flushed into a batch job on a schedule, and the results land where the interactive path would have put them. For a nightly pipeline that alone roughly halves the AI line on the bill.

6. Limits and budgets in one place

The expensive incidents in AI are loops: an agent that keeps calling a tool, a script that retries without backoff, a bot that floods a public chat. Put a gateway between your apps and the providers and give it hard limits: tokens per minute and a quota per key or per feature. In Azure API Management that is the llm-token-limit policy, and llm-emit-token-metric sends token usage with your own dimensions to Azure Monitor. The keys stay in the gateway, never in the apps.

Around it, the classic FinOps controls: budgets with forecast alerts in your cloud's cost management, provider spend limits where they exist, and the same short daily cost report in the team chat that works for cloud costs. Public endpoints need their own guard: the guide on this site allows 20 questions per visitor in ten minutes, which no real visitor reaches and every scraper does.

xml

<!-- Azure API Management, inbound: hard limits per key, usage per feature -->
<llm-token-limit counter-key="@(context.Subscription.Id)"
                 tokens-per-minute="20000"
                 token-quota="2000000" token-quota-period="Monthly"
                 estimate-prompt-tokens="false" />
<llm-emit-token-metric namespace="ai-usage">
  <dimension name="Feature" value="@(context.Request.Headers.GetValueOrDefault(&quot;x-feature&quot;, &quot;unknown&quot;))" />
  <dimension name="Subscription ID" />
</llm-emit-token-metric>

7. Know when self-hosting pays off

At some volume a per-token price loses to a machine you own: steady, predictable traffic, a model that is good enough in its open-weight version, and often data that shouldn't leave your network anyway. Below that volume, the API wins on cost, speed of change and the people you don't need to run it. The honest comparison includes power, operations and the time to keep models current, not only the hardware.

The tool below gives you the first number for that decision: pick a model size and how many people use it, and it shows the memory you need and the smallest machine that holds it. My article on which server to buy for a local LLM goes through the rest.

Interactive

Which device for a local LLM?

Pick a model and how you will use it. You get the memory it needs and the smallest machines that hold it, from a mini PC to a GPU server.

Start from

Model size (billion parameters)70B

Mixture-of-experts model

Precision

Who uses it

≈ 42 GB

memory needed

128 GB

memory in the smallest setup that fits

Smallest setup that fits

Mini PC with AMD Ryzen AI Max+ 395

The cheapest route to a lot of memory: up to 128 GB of 256-bit LPDDR5x-8000 (about 256 GB/s) shared with the Radeon 8060S graphics, 45 to 120 W. Runs llama.cpp, LM Studio and Ollama.

≤ 7 tokens/s, upper limit for one user on that machine

All options, roughly from small to large investment

PC with GeForce RTX 5090Too small
Fast for models that fit into 32 GB. A consumer card: loud, hot, no ECC memory, 575 W total graphics power.
Mini PC with AMD Ryzen AI Max+ 395Fits · 128 GB
The cheapest route to a lot of memory: up to 128 GB of 256-bit LPDDR5x-8000 (about 256 GB/s) shared with the Radeon 8060S graphics, 45 to 120 W. Runs llama.cpp, LM Studio and Ollama.
≤ 7 tokens/s, upper limit for one user on that machine
NVIDIA DGX SparkFits · 128 GB
GB10 superchip with 128 GB unified memory and the full CUDA stack. 273 GB/s limits large dense models; strong for mixture-of-experts models, development and fine-tuning experiments.
≤ 8 tokens/s, upper limit for one user on that machine
Mac Studio M5 MaxFits · 128 GB
Up to 128 GB unified memory at 460 to 614 GB/s, and quiet. MLX, llama.cpp, Ollama and LM Studio run well on it.
≤ 13–18 tokens/s, upper limit for one user on that machine
Workstation or server with RTX PRO 6000 Blackwell (1–2)Fits · 96 GB
96 GB ECC GDDR7 at 1,792 GB/s and 600 W per card: the fastest per user. One or two of them with vLLM make a server for a team.
≤ 51 tokens/s, upper limit for one user on that machine
Rack server with 2× L40SFits · 96 GB
Two 48 GB data-center GPUs with vLLM in front: the other team setup from the article for 30B to 70B models.
Mac Studio M5 UltraFits · 512 GB
Up to 512 GB unified memory at 1.2 TB/s, at most 480 W continuous: the simplest way to fit a very large model on a desk.
≤ 34 tokens/s, upper limit for one user on that machine
NVIDIA DGX Station (GB300)Fits · 748 GB
A deskside machine with the GB300 superchip and 748 GB of coherent memory, 252 GB of it HBM3e.
GPU server with H200 (up to 8)Fits · 141 GB
141 GB HBM3e at 4.8 TB/s and up to 700 W per GPU; eight of them give over a terabyte. vLLM splits the model across the GPUs.
NVIDIA DGX B200Fits · 1,440 GB
Eight B200 GPUs with 1,440 GB HBM3e and 64 TB/s in total, at about 14.3 kW for the system. Power, cooling and operations become the bigger project.

Planning estimate, not a benchmark. Memory = parameters × bytes per parameter, plus about 20% for the KV cache (long documents or a few people: +50%, a team: +100%, rough factors). Unified-memory machines keep about 10% for the system. The speed limit is memory bandwidth divided by the weights read per token (only the active ones for mixture-of-experts); real numbers are lower. Device figures from the vendors' spec pages, as listed in the article (Ryzen bandwidth calculated from AMD's 256-bit LPDDR5x-8000). Before buying, test your model and prompts on rented GPUs for two weeks.

Help choosing the hardware All calculators and checklists

Takeaway
AI FinOps is FinOps with tokens: attribute every call, shrink what you send, route to the smallest model that works, batch what can wait, and cap it all in one gateway.

Real projects behind this

Try it yourself

Want this in your company? Let's talk mail@dominikheinrich.ch

Sources and official guides