You estimated your LLM costs carefully. You looked at expected request volume, estimated average token counts, and built a model. Then you shipped, and the bill came in at three times your estimate. This is not a budgeting failure. It is a feature of how LLM pricing works.
Why token pricing creates variance by design
LLM providers charge per input token and per output token. That sounds simple, but the number of tokens in any given request is almost impossible to predict in advance because it depends entirely on what the user does.
Short queries cost a little. Long queries with lots of context cost a lot. A user who pastes an entire contract and asks for a detailed analysis will cost 50 times more than a user asking a quick question. Both are valid uses of your product. You cannot control which one any given user chooses to do.
This creates a fundamental asymmetry: your revenue per user is relatively predictable (subscription fee, per-seat price, usage tier), but your cost per user is not. That gap is where margins get destroyed.
The four things that actually cause spikes
Long conversations
Most LLM APIs require you to pass the full conversation history with each request. That means a 20-turn conversation costs roughly 20 times more per turn than a fresh query. Power users who explore deeply are your most engaged users. They are also your most expensive.
Complex queries
Certain types of requests generate much longer outputs. Summarisation requests, code generation, detailed explanations, and multi-step reasoning all produce more tokens. You cannot predict how many of these any given user session will contain.
Traffic spikes
A feature gets mentioned in a newsletter. A post goes viral. You get picked up by a major publication. Traffic doubles in 48 hours. On a fixed-cost infrastructure, a traffic spike is a good problem to have. On pay-per-token pricing, it can be a financial emergency.
Edge cases in your prompt
Certain inputs reliably cause models to generate unusually long outputs. Poorly specified instructions, ambiguous tasks, or inputs that trigger verbose model behaviour can all inflate your token count in ways that are hard to predict or catch in testing.
In our analysis of production LLM workloads, the top 10 percent of requests by cost account for more than 60 percent of total spend. The distribution is not normal. It has a very long tail.
What most teams try first (and why it does not work)
The typical response to an unexpected LLM bill is to add usage limits. Cap conversations at 10 turns. Truncate context after 2,000 tokens. Add a daily quota per user. These measures work in the sense that they reduce costs. They also reliably degrade your product and frustrate your best users.
The other common approach is to set a tighter max_tokens limit on responses. This helps at the margins but does not address the core problem, which is that you have no control over input length.
Approaches that actually work
There are a few strategies that address the variance problem without degrading the product.
Context window management means being intelligent about what you include in each request. Rather than passing the full conversation history, summarise older turns. Rather than including an entire document, extract the relevant sections. This requires engineering effort and ongoing maintenance as your product evolves.
Model routing means sending simpler requests to cheaper models automatically. A well-tuned routing layer can cut average cost per request by 40 to 60 percent. But it requires ongoing calibration and a monitoring system to catch quality regressions.
Absorbing the variance at the infrastructure layer is the approach that makes costs genuinely predictable. Instead of trying to control every input, you put a layer between your application and the LLM provider that handles optimisation, absorbs spikes, and charges a fixed fee regardless of what your users do.
The first two approaches require your team to build and maintain infrastructure that is not your core product. The third approach is what Griyo provides.