How to reduce LLM API costs: caching, batching and model routing

Key takeaways

• To reduce LLM API costs, measure first: log input, cached, output and thinking tokens for every request, then price them per feature.
• Prompt caching is the biggest lever for apps with long, repeated prompts. Cache reads cost 0.1x the normal input price or less at OpenAI and Anthropic, according to their documentation.
• Batch APIs at OpenAI, Anthropic and Google charge 50% of standard prices for work that can wait up to 24 hours.
• Routing easy tasks to a small model and capping output length often saves more than any single provider discount.
• In our worked example (prices as of Oct. 23, 2026), the four steps cut a $1,900 monthly bill to about $198, a fall of roughly 90%. Your result will depend on traffic and prompt shape.

You can reduce LLM API costs by stacking four techniques: prompt caching, batch APIs, model routing and output limits. In a worked example, they cut a $1,900 monthly bill to about $198, using official OpenAI, Anthropic and Google prices as of Oct. 23, 2026.

The examples use the official Python SDKs (openai, anthropic and google-genai). The ideas carry over to any provider. Prices change often, and Google's Gemini rates are promotional until the end of 2026, so check each pricing page before you rely on the numbers below.

Where LLM API costs come from: input vs output tokens

Every provider bills per million tokens, and each request has several parts that are priced differently:

• Uncached input: the system prompt, documents, chat history and user message, billed at the base input price.
• Cache writes: the first time a long prefix is stored. Anthropic charges 1.25x the base input price for a 5-minute cache and 2x for a 1-hour cache. On OpenAI's GPT-5.6 and later models, a write costs 1.25x with the default 30-minute lifetime and 2.5x with the 12-hour option.
• Cache reads: later requests that reuse the stored prefix, billed at a fraction of the input price.
• Output: the generated text, which usually costs four to six times as much per token as input. Reasoning or "thinking" tokens are billed as output too, even when you only see a summary of them.

Here are the published list prices for the models used in this guide:

USD per 1 million tokens, standard short-context rates. Data as of 2026-10-23 07:30 UTC. Sources: OpenAI API pricing, Anthropic pricing, Google Gemini API pricing (links below). Gemini 3.8 Flash doubles to $1.50 / $7.50 from Jan. 1, 2027, and Gemini 4 Argon to $4 / $20 from Dec. 1, 2026.

Two details catch people out. Anthropic's Claude Haiku 5.5 rises to $0.50 input and $2.50 output for prompts over 100,000 tokens. OpenAI charges more for prompts over 272,000 tokens; gpt-6-astra, for example, goes from $10 to $20 per million input tokens. Long prompts cost more per token as well as in total. For a wider model comparison, see our Gemini 4 Argon benchmarks and pricing comparison.

Step 1: Measure token spend per request (Python)

You cannot cut what you have not measured. The helper below turns each provider's usage object into the same four numbers, prices them, and appends one line per call to a JSONL file. Tag each call with a feature name so you can see which part of your product is expensive.

import json, time

# USD per 1M tokens, from the official pricing pages (as of Oct. 23, 2026)
PRICES = {
    "claude-sonnet-5-5": {"input": 2.00, "write": 2.50,  "read": 0.10,  "output": 10.00},
    "claude-haiku-5-5":  {"input": 0.10, "write": 0.125, "read": 0.01,  "output": 0.50},
    "gpt-6.1-sol":       {"input": 2.00, "write": 2.50,  "read": 0.10,  "output": 10.00},
    "gpt-6-luna":        {"input": 0.10, "write": 0.125, "read": 0.01,  "output": 0.50},
    "gemini-3.8-flash":  {"input": 0.75, "write": 0.75,  "read": 0.075, "output": 3.75},
}

def cost_usd(model, uncached, written, read, output):
    p = PRICES[model]
    return (uncached * p["input"] + written * p["write"]
            + read * p["read"] + output * p["output"]) / 1_000_000

def from_anthropic(resp):
    u = resp.usage  # total input = input_tokens + cache_creation + cache_read
    return dict(uncached=u.input_tokens,
                written=u.cache_creation_input_tokens or 0,
                read=u.cache_read_input_tokens or 0,
                output=u.output_tokens)

def from_openai(resp):
    u = resp.usage
    d = u.input_tokens_details
    read = d.cached_tokens or 0
    written = getattr(d, "cache_write_tokens", 0) or 0
    return dict(uncached=u.input_tokens - read - written,
                written=written, read=read,
                output=u.output_tokens)  # includes reasoning tokens

def from_gemini(interaction):
    u = interaction.usage
    read = u.total_cached_tokens or 0
    return dict(uncached=u.total_input_tokens - read, written=0, read=read,
                output=(u.total_output_tokens or 0) + (u.total_thought_tokens or 0))

def log_call(model, feature, tokens, path="llm_costs.jsonl"):
    row = {"ts": time.time(), "model": model, "feature": feature, **tokens,
           "usd": round(cost_usd(model, **tokens), 6)}
    with open(path, "a") as f:
        f.write(json.dumps(row) + "\n")
    return row

Check the helper against your provider's billing dashboard for a day before trusting it. The field names come from each provider's documentation. Anthropic says total input equals input_tokens plus cache_creation_input_tokens plus cache_read_input_tokens. OpenAI reports cache reads in input_tokens_details.cached_tokens and, on GPT-5.6 and later, writes in cache_write_tokens. Google's Interactions API reports total_cached_tokens and total_thought_tokens. Our Gemini 4 Argon API tutorial explains why thinking tokens count as output.

After a few days of logs, group by feature and model. Most teams find that one or two features account for most of the bill. Start with those.

Step 2: Prompt caching at OpenAI, Anthropic and Google

Prompt caching stores the processed form of a prompt prefix so later requests that start with exactly the same text are cheaper. The rule is the same at all three providers: put stable content first and variable content last. System instructions, tool definitions, policy documents and few-shot examples go at the top; the user's message goes at the end. If anything early in the prompt changes, even a timestamp, the cache cannot be used for anything after it.

Data as of 2026-10-23 07:30 UTC. Sources: OpenAI prompt caching guide, Anthropic prompt caching documentation, Google Gemini context caching and pricing pages.

Anthropic: mark the stable block

import anthropic

client = anthropic.Anthropic()
POLICY = open("support_policy.md").read()  # ~6,000 tokens, identical on every call

def answer(ticket_text):
    resp = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=400,
        system=[
            {"type": "text", "text": "You are a support assistant. Answer in at most 4 sentences."},
            {"type": "text", "text": POLICY, "cache_control": {"type": "ephemeral"}},
        ],
        messages=[{"role": "user", "content": ticket_text}],
    )
    log_call("claude-sonnet-5-5", "support", from_anthropic(resp))
    return resp.content[0].text

The first call writes the cache at 1.25x the input price. Calls within the next five minutes read it at a fraction of that price, and each hit resets the five-minute timer at no extra cost. Anthropic allows up to four explicit breakpoints per request. If traffic is sparse, the documented "ttl": "1h" option keeps the entry longer, but each write then costs 2x the base input price.

OpenAI: keep the prefix identical

from openai import OpenAI

client = OpenAI()

resp = client.responses.create(
    model="gpt-6.1-sol",
    instructions=STATIC_INSTRUCTIONS + POLICY,  # stable prefix first
    input=ticket_text,                          # variable part last
    max_output_tokens=400,
    prompt_cache_key="support-bot",             # optional: separate cache accounting per feature
    # prompt_cache_options={"ttl": "12h"},      # longer life, 2.5x write price
)
log_call("gpt-6.1-sol", "support", from_openai(resp))

OpenAI caches automatically, so the main work is keeping the prefix byte-for-byte identical. On GPT-5.6 and later, the docs say prompt_cache_key is optional and is used for separate cache accounting, for example per customer. Cached tokens still count toward tokens-per-minute rate limits.

Google Gemini: let implicit caching work

Gemini 4 Argon supports only implicit caching, and Google's caching page says explicit caching is not available for it or through the Interactions API. Savings apply automatically once a repeated prefix passes 4,096 tokens. Check usage.total_cached_tokens to confirm hits. Explicit caches for older Gemini 3 and 2.5 models add an hourly storage fee; Google lists $0.50 per million tokens per hour for Gemini 3.8 Flash through Dec. 31, 2026, so they pay off only when a large context is reused often.

Step 3: Batch APIs, when 50% off is worth the wait

All three providers sell asynchronous batch processing at half the standard price. You give up an instant answer. In return you get a lower bill and, at OpenAI, a separate rate-limit pool.

• OpenAI Batch API: a JSONL file of up to 50,000 requests (200 MB), each with custom_id, method, url and body. Completion window is 24 hours.
• Anthropic Message Batches: up to 100,000 requests or 256 MB. Anthropic says most batches finish within an hour; unfinished ones expire after 24 hours, and results stay downloadable for 29 days.
• Gemini Batch API: priced at 50% of the interactive cost, with a 24-hour target turnaround. Inline requests must stay under 20 MB; input files can be up to 2 GB.

Good candidates are nightly summaries, document tagging, evaluation runs, embedding backfills and anything a person will not wait on. Here is a complete Anthropic batch job that also uses a one-hour cache, which Anthropic recommends for batches that share context because processing can take longer than five minutes:

import time
import anthropic

client = anthropic.Anthropic()

batch = client.messages.batches.create(requests=[
    {"custom_id": f"doc-{i}",
     "params": {
         "model": "claude-sonnet-5-5",
         "max_tokens": 300,
         "system": [{"type": "text", "text": POLICY,
                     "cache_control": {"type": "ephemeral", "ttl": "1h"}}],
         "messages": [{"role": "user", "content": doc}],
     }}
    for i, doc in enumerate(docs)
])

while client.messages.batches.retrieve(batch.id).processing_status != "ended":
    time.sleep(60)

for r in client.messages.batches.results(batch.id):
    if r.result.type == "succeeded":
        print(r.custom_id, r.result.message.content[0].text)
    else:
        print("retry later:", r.custom_id, r.result.type)

Anthropic's pricing page says batch and prompt-caching discounts stack. OpenAI's batch price list also includes discounted cached-input rates. The OpenAI version of the job writes the same requests to a .jsonl file, uploads it with purpose="batch" and calls client.batches.create(input_file_id=..., endpoint="/v1/responses", completion_window="24h"). Results can come back in any order, so always match on custom_id.

Step 4: Model routing, sending easy tasks to small models

The price gap between tiers is large. Claude Haiku 5.5 lists at $0.10 per million input tokens against $2 for Sonnet 5.5, and gpt-6-luna at $0.10 against $2 for gpt-6.1-sol. Classification, extraction, tagging and short summaries rarely need a frontier model.

A router does not need to be clever. Start with rules, and escalate only when the cheap answer fails a check:

SMALL, LARGE = "claude-haiku-5-5", "claude-sonnet-5-5"
EASY_TASKS = {"classify", "extract", "tag", "short_summary"}

def pick_model(task, text):
    if task in EASY_TASKS and len(text) < 20_000:
        return SMALL
    return LARGE

def run(task, text, validate):
    model = pick_model(task, text)
    out = call_model(model, task, text)      # your wrapper around messages.create
    if model == SMALL and not validate(out):
        out = call_model(LARGE, task, text)  # escalate only on failure
    return out

The validate function is where the savings are kept safe. It can check that the output parses as JSON, that a label is in the allowed set, or that required fields are present. Log how often escalation happens. If the small model fails more than about one time in ten on a task, move that task to the large tier, because paying twice for the same request cancels the saving. For work that can stay entirely on your own hardware, a model served through Ollama removes the per-token bill, though not the hardware and power costs.

Step 5: Trim context and cap output length

Output tokens are the most expensive tokens, so set a hard limit on every call: max_tokens for Anthropic, max_output_tokens for OpenAI's Responses API, and max_output_tokens inside generation_config for Gemini. Back the limit up with instructions such as "answer in at most four sentences" or a fixed JSON schema. A cap alone can cut answers off mid-sentence. On reasoning models, thinking counts toward the limit, so a cap that is too low can still bill you for thinking and return no usable answer. Our prompt engineering guide covers ways to get short, structured replies.

On the input side, chat history and tool results grow with every turn. In agent loops like the one in our tool-calling agent tutorial, they often become the largest part of the bill. Trim from the middle, not the front, so the cached prefix survives:

def trim_history(messages, keep_last=6):
    """Keep the opening task message and the most recent turns."""
    if len(messages) <= keep_last + 1:
        return messages
    return messages[:1] + messages[-keep_last:]

Check that the trimmed list still alternates user and assistant turns, which the Anthropic Messages API requires. Other easy wins: send retrieved passages instead of whole documents, shorten bulky tool outputs before returning them to the model, and keep prompts under the long-context price thresholds mentioned above.

Worked example: before/after cost table

Take a support assistant that handles 100,000 requests a month on Claude Sonnet 5.5. Each request carries a 6,000-token policy prefix that never changes, 500 tokens of user input, and about 600 tokens of output. Our assumptions: 2% of requests write the cache and 98% read it; output caps bring the average answer down to 350 tokens; 60% of tickets are simple enough for Claude Haiku 5.5; and 30% of the total workload, such as overnight ticket tagging, can move to the Batch API at the same cache hit rate. Prices are Anthropic's list prices as of 2026-10-23 07:30 UTC.

Bullquill calculation from Anthropic list prices (data as of 2026-10-23 07:30 UTC). Usage assumptions are illustrative. Routing assumes no escalations, and the batch row ignores the higher 2x price of one-hour cache writes.

The order matters. Caching did the most because the prompt was mostly a repeated prefix. Routing came second because the gap between Sonnet 5.5 and Haiku 5.5 is twentyfold. Batching added the least here because it applied to only part of the traffic. An app with short, unique prompts would see little from caching and more from routing and output caps, which is why Step 1 comes first.

What it means for developers

List prices are falling. Anthropic cut Opus pricing with version 5.5, and Google is running introductory Gemini rates through the end of 2026. Even so, the way an app is built still matters more than which provider it uses. A badly structured prompt on a cheap model can cost more than a well-cached prompt on an expensive one. The habits that last are the ones in this guide: log every call, keep prompts stable at the front, push non-urgent work into batches, use the smallest model that passes your checks, and cap output.

Two cautions. First, Google's promotional Gemini prices double on Dec. 1, 2026 (Gemini 4 Argon) and Jan. 1, 2027 (Gemini 3.8 Flash), so budgets built on today's rates need a review date. Second, discounts have conditions: minimum prompt lengths, cache lifetimes, and batch jobs that can expire. Re-run your cost logs after every model or pricing change rather than assuming the savings hold.

Sources

• OpenAI: Prompt caching guide
• OpenAI: API pricing
• OpenAI: Batch API guide
• Anthropic: Prompt caching
• Anthropic: Pricing
• Anthropic: Batch processing (Message Batches API)
• Google: Gemini API context caching
• Google: Gemini Developer API pricing
• Google: Gemini Batch API
• Google: Interactions API reference