LLM API costs have a way of creeping up on you. You launch with a rough estimate, usage grows, and three months later you are looking at a bill that makes no sense. The instinct is to start throttling users or degrading responses. That is the wrong move.
Most of the waste in LLM API spend comes from the infrastructure layer, not the product layer. That means you can recover a significant chunk of it without your users ever noticing.
Prompt compression
The single biggest lever for most teams is prompt length. LLM providers charge by the token, and the majority of tokens in a typical request are in the prompt, not the response. Long system prompts, verbose instructions, and bloated context windows all add up.
Start by auditing your system prompts. Most system prompts grow organically over time and contain redundant instructions, examples that are no longer representative, and caveats that never get triggered. A prompt audit typically finds 20 to 40 percent of tokens that can be removed without changing output quality at all.
For context that changes between requests, be specific about what you include. If a user has a 10,000 word document but only asks about the pricing section, summarise or extract the relevant part before passing it to the model. You do not need to include everything every time.
A well-run prompt audit typically reduces input token usage by 25 to 35 percent. On a $10,000 monthly bill, that is $2,500 to $3,500 per month with no product changes.
Semantic caching
Caching LLM responses is not the same as caching a database query, but the principle is similar. If two users ask functionally the same question, you should not be paying for two completions.
Semantic caching works by embedding incoming requests and checking for high-similarity matches in a vector store. When a match is found above a confidence threshold, you return the cached response instead of hitting the API. For most production applications, cache hit rates of 20 to 35 percent are achievable within the first few weeks.
The key is setting the right similarity threshold. Too tight and you miss obvious matches. Too loose and you return irrelevant responses. Start conservative at 0.95 cosine similarity and loosen gradually as you build confidence.
Model routing
Not every request needs your most powerful model. A user asking for a one-sentence summary of a paragraph does not need GPT-5.5. A user building a complex multi-step reasoning chain probably does.
Model routing means classifying requests by complexity before they reach your LLM and routing simpler tasks to cheaper, faster models. A well-tuned routing layer can cut your average cost per request by 40 to 60 percent while keeping quality high on the tasks that actually need it.
Common routing signals include: request length, presence of specific keywords, task type classification, and user tier. Start with a simple rule-based router and refine it with a small classifier model if you need more precision.
Batching and async processing
For tasks that do not need real-time responses, batching requests is one of the simplest ways to cut costs. OpenAI's batch API, for example, offers a 50 percent discount on processing jobs that can tolerate up to 24 hour turnaround.
Common candidates for batching include: document processing pipelines, background summarisation, nightly report generation, and email drafts that users queue rather than send immediately. If the user does not need an answer in under two seconds, consider whether it needs to be a live API call at all.
Output length control
Most models are verbose by default. They hedge, they summarise, they add caveats. All of that costs tokens. If your application only needs a structured JSON output or a short answer, tell the model explicitly and set a max token limit on the response.
A max_tokens parameter that is 30 percent tighter than your current average response length will rarely cause truncation on valid responses but will prevent the occasional runaway generation that inflates your bill.
Putting it together
These techniques compound. A 25 percent reduction from prompt compression, combined with a 25 percent cache hit rate and 40 percent of traffic routed to cheaper models, can result in an overall cost reduction of 50 to 65 percent on a typical production workload.
The challenge is that all of this takes engineering time and ongoing maintenance. The routing layer needs to be tuned. The cache needs to be monitored. The prompts need regular auditing as the product evolves.
If you want the savings without building the infrastructure yourself, that is exactly what Griyo does. We absorb the optimisation layer and charge a single fixed monthly fee, so your costs become predictable regardless of usage patterns.