6 Techniques To Reduce LLM API Costs With the Python Library

Before diving into solutions, it helps to understand the scale of the problem.

Take a common production pattern: a customer support bot that processes 10,000 messages per day, each with a 2,000-token system prompt and a 200-token user message.

Cost breakdown by the numbers:

That's not a benchmark; it's arithmetic. The techniques don't require magic; they require applying what providers already offer.

Technique 1: Prompt Caching, Up to 90% Off Repeated Content

The Problem

Most LLM applications send the same system prompt on every request. If your system prompt is 2,000 tokens and you make 10,000 requests per day, you're paying for 20 million input tokens daily, even though the content never changes.

How It Works

Anthropic's prompt caching lets you mark content blocks with cache_control. The first request pays full price and writes to cache. Every subsequent request that hits the same cached content pays 10% of the normal input price. Cache entries last 5 minutes and reset on each hit.

The key insight: cache the most stable content first. Your base instructions change rarely. Your few-shot examples change occasionally. Your per-request context changes every time. Structure your prompt from most stable to least stable.

from llm_optimizer import OptimizedClient, build_cached_system_prompt
import anthropic

client = OptimizedClient(anthropic_client=anthropic.Anthropic())

# Build an optimally structured cached system prompt
system = client.build_cached_system(
    base_instructions="""
    You are an expert customer support agent for a SaaS company.
    You have deep knowledge of our product, billing, and technical issues.
    Always be empathetic, clear, and solution-focused.
    [... 1,500 more tokens of stable instructions ...]
    """,  # ← cached after first request — 10% cost on all subsequent calls

    few_shot_examples="""
    Example 1: Billing question → here's how to handle it
    Example 2: Technical issue → here's the escalation path
    [... 500 tokens of examples ...]
    """,  # ← also cached separately
)

# First call: pays full price, writes to cache
response1 = client.complete(messages=[{"role": "user", "content": "How do I cancel?"}], system=system)

# Second+ calls: system prompt served from cache at 10% cost
response2 = client.complete(messages=[{"role": "user", "content": "Where's my invoice?"}], system=system)

What the Library Does

llm-optimizer automatically injects cache_control breakpoints at optimal positions, system prompt, few-shot examples, and long conversation history, respecting Anthropic's 4-breakpoint limit. You don't touch the API directly.

Savings Calculation

2,000 token system prompt × 10,000 requests/day = 20M tokens/day
Without caching: 20M × $3.00/1M (Sonnet)  = $60.00/day
With caching:    2M × $3.00 + 18M × $0.30 = $11.40/day
Savings: $48.60/day = $17,739/year

Technique 2: Model Routing, 60% to 80% Off by Using the Right Model

The Problem

Routing every request to your best model is the most common and most expensive mistake. Claude Opus costs 5x more than Claude Haiku. For tasks that Haiku handles perfectly, such as classification, extraction, translation, and simple Q&A, you're paying a 500% premium for no benefit.

The Naive Approach and Why It Fails

The obvious solution is to route by keyword: if the prompt contains "classify," use Haiku; if it contains "analyze," use Sonnet. This works until it doesn't.

A prompt like "Explain the constitutional implications of this clause" is 8 words. Short, simple-looking. A keyword router sees no complexity signals and routes it to Haiku. But the task requires expert-level legal reasoning. This class of error intent-heavy short prompts is the primary failure mode of heuristic routing.

The Better Approach: Use a Classifier

llm-optimizer solves this by optionally using Haiku itself to classify task complexity before routing. The cost is approximately 15 tokens,  about $0.000015. If that classification prevents one wrong Opus call (2,000 tokens × $5/1M = $0.01), it pays for itself 666 times over.

from llm_optimizer import OptimizedClient, Provider

client = OptimizedClient(
    anthropic_client=anthropic.Anthropic(),
    enable_llm_classifier=True,  # uses Haiku to assess complexity — ~$0.000015/call
    preferred_provider=Provider.ANTHROPIC,
)

# Short prompt, complex intent → correctly routed to Opus
response = client.complete(
    messages=[{"role": "user", "content": "Explain the constitutional implications of this clause"}]
)

# You can audit the routing decision
from llm_optimizer import ModelRouter
router = ModelRouter(enable_llm_classifier=False)
print(router.explain("classify this email as spam or not"))
# {
#   "detected_complexity": "simple",
#   "routed_model": "claude-haiku-4-5",
#   "keyword_signals_fired": {"simple": ["classify"]},
#   "token_count": 8
# }

Complexity Tiers

A closer look at complexity:

1,000 requests/day — mixed complexity
Without routing: all → Opus ($5/1M input)
  1,000 × 500 tokens = 500K tokens × $5 = $2.50/day

With routing: 70% Haiku, 20% Sonnet, 10% Opus
  700 × 500 × $1 + 200 × 500 × $3 + 100 × 500 × $5 = $1.05/day

Savings: 58% reduction

Technique 3: Prompt Optimization, 5% to 20% Off Token Count

The Problem

Prompts written by humans, especially in collaborative or enterprise settings,  accumulate filler. Phrases like "please note that", "it is important to note that", "in order to", and "due to the fact that" add tokens without adding meaning. At scale, this is a measurable cost.

What the Library Strips

from llm_optimizer import PromptOptimizer

opt = PromptOptimizer()
result = opt.optimize("""
    In order to complete this task, please note that you should carefully
    analyze the following text. It is important to note that accuracy
    matters.   Please be aware that   your response should be concise.
""")

print(result.optimized_text)
# "To complete this task, carefully analyze the following text.
#  Accuracy matters. Your response should be concise."

print(f"Saved {result.tokens_saved} tokens ({result.savings_pct}%)")
# Saved 18 tokens (31%)

What is never touched: Code blocks, factual content, user-specified phrasing. The optimizer is conservative by default. It only removes patterns with no semantic value.

Conversation history trimming: In long conversations, the library keeps the last N turns and drops older context, preventing unbounded token grow.

Technique 4: Batch Processing, 50% Off Non-Urgent Requests

The Problem

Not every LLM call needs an immediate response. Nightly report generation, document indexing, data enrichment pipelines, and offline classification jobs all of these run fine with a delay. But most teams send them as real-time requests anyway, paying full price.

How Anthropic's Batch API Works

Anthropic's Message Batch API processes up to 10,000 requests per batch at 50% of the normal price. Results are available within minutes to hours. The trade-off is explicit: cost for latency.

client = OptimizedClient(
    anthropic_client=anthropic.Anthropic(),
    enable_batching=True,
)

# Queue 1,000 document summaries throughout the day
for doc in documents:
    client.queue(
        custom_id=doc["id"],
        messages=[{"role": "user", "content": f"Summarize: {doc['text']}"}],
        max_tokens=200,
    )

# Submit as one batch — 50% cheaper than 1,000 individual calls
batch_id = client.submit_batch()

# Poll when ready — minutes to hours depending on load
results = client.poll_batch(batch_id, wait=True)
for r in results:
    print(f"{r.custom_id}: {r.content}")

When to Use It

• Nightly data processing pipelines
• Document indexing and enrichment 
• Offline classification and tagging 
• Report generation 

Technique 5: Document Compression to Reduce Context Before Sending

The Honest Tradeoff

This technique requires a direct warning: document compression is lossy. Removing content from a document to reduce token count means the model works with less information. For some tasks this is fine; for others it produces wrong answers.

Use it only when:

• You've verified empirically that compression doesn't hurt your answer quality.
• You're doing rough extraction where completeness isn't required.
• You have re-ranking downstream (e.g., RAG pipelines).

Do not use it for legal documents, compliance reviews, or any task where every sentence may be relevant.

TF-IDF Extractive Compression

When you do use compression, naive truncation (cutting from the end) is the worst strategy. llm-optimizer implements TF-IDF paragraph scoring where each paragraph is scored by its term overlap with your query, weighted by how unique those terms are across the document. The most relevant paragraphs fill the token budget; the rest are dropped.

from llm_optimizer import DocumentCompressor

# ⚠️ Read the accuracy warning before using in production
comp = DocumentCompressor(
    max_tokens=4000,
    strategy="extractive",  # TF-IDF scoring — best accuracy
)

compressed, tokens_saved = comp.compress(
    document=long_contract,      # 50,000 tokens
    query="payment terms and termination clauses"  # focus compression here
)

print(f"Compressed to {4000} tokens, saved {tokens_saved} tokens")
# All compressed output includes a visible [⚠️ COMPRESSION WARNING] marker

There are three strategies available: 

Strategy options:

Technique 6: Cost Tracking That Observes Before You Optimize

Why This Matters

You can't optimize what you don't measure. Before applying any of the above techniques, you need to know:

• Which models you're actually using
• 
  
• Where your token spend is going
• Whether your optimizations are working

client = OptimizedClient(
    anthropic_client=anthropic.Anthropic(),
    persist_tracking="usage.jsonl",  # survives restarts
)

# ... run your application ...

client.print_summary()
# ═══════════════════════════════════════════════════════
#   LLM Cost Optimizer — Usage Summary
# ═══════════════════════════════════════════════════════
#   Total Requests   : 1,247
#   Total Cost       : $0.8432
#   Total Saved      : $7.2180  (89.5% savings)
#   Cached Tokens    : 8,432,000
#   By Model         : haiku: 891 reqs ($0.12) | sonnet: 312 ($0.58)
#   Optimizations    : prompt_caching: 1247x | model_routing: 1247x
# ═══════════════════════════════════════════════════════

The tracker records every request's tokens, cost, cached tokens, savings, latency, and which optimizations fired. Data persists to JSONL so you can analyze it across sessions or pipe it to your observability stack.

Architecture: Why Not Just Use LiteLLM?

The obvious question. LiteLLM is excellent and covers a lot of ground, including unified provider API, routing, cost tracking, batch processing. If you're not already using it, you should evaluate it.

llm-optimizer does three things LiteLLM doesn't:

1. Automatic cache_control injection: LiteLLM passes caching headers through but doesn't inject breakpoints at optimal positions automatically.
2. Prompt filler stripping: LiteLLM has no token-level prompt optimization.
3. TF-IDF document compression: LiteLLM has no query-aware document compression.

The intended use is actually as a complement: llm-optimizer can sit on top of a LiteLLM setup, handling the prompt-level optimizations that LiteLLM doesn't touch.

Streaming Support

For user-facing applications, the library supports streaming:

with client.stream(
    messages=[{"role": "user", "content": "Explain quantum entanglement"}],
    system="You are a physics tutor.",
    max_tokens=512,
) as stream:
    for chunk in stream:
        print(chunk, end="", flush=True)

# Access token usage after stream completes
usage = stream.usage()

All optimizations, including caching, routing, prompt, and optimization, apply identically to streaming requests.

Error Handling

Production LLM applications need to handle rate limits and model overloads gracefully. The library handles this automatically:

client = OptimizedClient(
    anthropic_client=anthropic.Anthropic(),
    max_retries=3,        # retry on rate limit with exponential backoff
    retry_base_delay=1.0, # 1s, 2s, 4s
)
# Rate limit (429) → retried with backoff
# Model overloaded (529) → falls back to next capable model automatically
# Non-retriable error → raises immediately

Limitations

Honest about what this doesn't do yet:

• Not org-scale validated: v0.4.0 is tested against ai-core Bedrock and Anthropic direct (14 live tests, all passing). Not yet run against production workloads at team scale. The pilot measures this.
• No async support: complete() and stream() are synchronous. Async support planned for a future release.
• OpenAI and Google partially tested: Anthropic and AWS Bedrock are the validated providers. OpenAI is implemented but not end-to-end tested in CI. Google Gemini streaming is not yet implemented.
• Token counting is approximate: The default estimator is within ~20% of the actual count. Install tiktoken for exact counts: pip install llm-optimizer[tiktoken].
• Pricing data can go stale: Stored in pricing.json with a version stamp. The library warns automatically if data is older than 30 days.
• Model allowlist: Only models listed in pricing.json can be routed to. Mythos and Fable 5 are not in the registry and cannot be called. Adding a new model requires a deliberate update to pricing.json.

# Basic install
pip install llm-optimizer

# With exact token counting
pip install llm-optimizer[tiktoken]

# All providers
pip install llm-optimizer[all]

import anthropic
from llm_optimizer import OptimizedClient

client = OptimizedClient(
    anthropic_client=anthropic.Anthropic(),
    # All optimizations on by default except compression (lossy — opt-in)
)

response = client.complete(
    messages=[{"role": "user", "content": "Your prompt here"}],
    system="Your system prompt here",
)

client.print_summary()

Links:

• PyPI: https://pypi.org/project/llm-optimize
• GitHub: https://github.com/banerjeeso/llm-optimiz

What's Next

• Async support (acomplete(), astream())
• LiteLLM adapter
• Budget guard: raise before a request exceeds a cost threshold
• Real production benchmarks once I've run this against a live workload.

Feedback welcome, especially from anyone who works with LLM APIs in production and can stress-test the routing logic or compression accuracy.

Published on PyPI as llm-optimizer. MIT license. Contributions welcome.