How to Reduce LLM Costs Without Losing Useful Answers
Separate context selection, provider caching, output limits, model choice and application caching. Measure savings against answer quality.
An increasing LLM bill needs an explanation before it needs a new model. Break usage down by route, model, input, output, cache behavior, retries and tool calls. More users, longer conversations and repeated failed requests can produce similar invoice changes while requiring different fixes.
Start with cost per successfully completed task. A cheaper call that fails and triggers several retries can cost more than the original path. Keep a small set of representative questions and expected outcomes so cost changes can be evaluated alongside quality.
Count the context that reaches the model
Inspect the actual request, including system instructions, tool definitions, history and retrieved documents. Avoid assuming the visible user question represents most of the input. Record these components separately where the provider's usage data allows it.
Select context that supports the current task. For a code change, that may include a function, its callers, relevant types and project conventions. For a support answer, it may include the current policy and earlier troubleshooting. Do not remove a condition or exception merely because it is expensive to include.
The context-selection study by Kabongo and colleagues evaluates leaderboard extraction from research papers. Its task-specific results illustrate why the selected evidence matters. Use your own failure cases to decide what can be omitted safely.
Moving text into a system message does not make it free
A system message establishes instruction priority and behavior. It does not by itself remove those tokens from the request's billable input. Keep necessary instructions, eliminate genuine duplication, and measure usage after the change.
A provider's caching mechanism may charge repeated input differently. That is a separate feature with its own eligibility, lifetime and billing rules. Check usage records to see whether repeated input actually received a cache discount.
Distinguish three kinds of caching
| Mechanism | What is reused | Important boundary |
|---|---|---|
| Provider prompt caching | Processing of eligible repeated input | Provider-specific matching, expiry and prices |
| Application result caching | An earlier answer or computed value | Scope, freshness and permission checks |
| Retrieval caching | Embeddings or selected evidence | Model/configuration versions and source updates |
Anthropic prompt caching works with prompt prefixes and cache controls. Gemini caching distinguishes implicit and explicit caching; explicit cached content uses a resource reference. Follow the interface for the provider and model actually deployed.
Check cache reads, writes and any storage charges in usage records. A cache that expires before reuse can add work without delivering the expected benefit. A result cache is a different decision because it can return an old answer without consulting a model or updated evidence at all.
For shared applications, include the relevant tenant, permissions and source version in the cache design. Never reuse a personalized answer across users simply because their questions look similar.
Use a transparent cost example
Suppose a fictional workload makes 100,000 requests per month. Removing 2,000 unnecessary input tokens from each request removes 200 million input tokens. At an assumed rate of $2 per million, the gross difference is $400.
The calculation assumes a uniform rate, no cache discount and no compensating increase in output, retrieval or retries. If an added retrieval service costs $150 for that workload, the remaining difference is $250 before other changes. Keep the same quality criterion on both sides.
This is why “we cut context by 90%” and “we cut the invoice by 90%” are different claims. The memory operating-cost guide expands the workload model.
Control output without mistaking format for correctness
Ask for the fields the application needs and apply an appropriate output limit. Structured output can simplify parsing, but JSON is not necessarily shorter than prose. Repeated keys, optional fields and verbose explanations can increase tokens.
Schema-constrained output can improve format compliance where supported. It does not prove that a value is true or that the evidence supports it. Validate business rules and source references separately, and handle refusals or incomplete output according to the API's documented behavior.
Use deterministic code for deterministic work such as arithmetic, known-format parsing and exact validation. Reserve model calls for tasks where language interpretation or generation contributes useful value.
Compare models and deployment choices fairly
Try a smaller or less expensive model on a fixed evaluation set. Compare task success, retries, escalation rate and latency as well as the rate card. Route only the cases that meet your acceptance criteria; one difficult task should not force every request onto the same model.
Fine-tuning, self-hosting and quantization are separate choices. Each changes the compute requirements, operating work and quality checks. Estimate compute utilization, operations, data preparation and evaluation work for the actual model and workload before deciding whether a managed API or your own deployment is cheaper.
For work that can wait, evaluate the provider's supported batch interface and its current terms. Application-side batching, serverless hosting and a discounted model-provider batch API are not interchangeable concepts.
Add memory only when it helps the task
Persistent memory can provide selected context from earlier interactions. Include its ingestion, retrieval and operating costs in the comparison. Test whether it preserves the facts that matter with fewer unnecessary tokens.
Keep original source records where your retention policy permits them. A concise memory that loses a date, condition or correction can produce an expensive business mistake despite making the prompt smaller.
Evaluate Supermemory on one recurring workflow and compare it with your existing context path. Report the observed quality, context size, service usage and total cost together.
Frequently asked questions
Are system messages free or billed only once?
No. Moving instructions into a system message does not itself remove input-token charges. Caching behavior and pricing depend on the provider and request.
Does prompt caching always use a cache ID?
No. Providers use different mechanisms. Some reuse matching prompt prefixes; explicit cached-content resources are another interface.
Does structured JSON always reduce cost?
No. It can improve parsing but may be longer than a concise text answer. Measure tokens and validate meaning separately.