How We Cut Our LLM Summarizer's Cost And Latency While Protecting Quality

We run an LLM pipeline that summarizes support tickets for a large network operations team: it reads long, noisy ticket logs and produces short summaries, customer status notifications, technician ticket summaries, and regional executive rollups. It works. But once it was running steadily, a quieter problem showed up. On an ordinary day, summarizing every ticket with a top-tier model and a full log was more expensive and slower than it needed to be.

Four levers — incremental summarization, trimming, right-sizing, and parallel execution — are compounding to cut blended LLM cost by roughly 60%.

Keeping the pipeline alive during traffic spikes is a separate story. This one is about the bill and latency on a normal Tuesday, and the four changes that, so far, are cutting our LLM cost by roughly 60% and making summaries faster while protecting quality.

Where the Cost and the Milliseconds Actually Go

Before optimizing anything, it helps to look at the anatomy of a single summarization call. It costs you two things — tokens and time — and both come from the same handful of places:

• Input tokens: The ticket log, plus the context we retrieve from our vector store, plus the prompt. For a summarizer, this is the giant. Logs are long; the summary is a few sentences.
• Output tokens: The summary itself is small, but output tokens are the most expensive per token, and generation time scales with how many you produce.
• The model: Price per token varies enormously between a flagship model and a smaller one.
• Round-trips: Doing calls one at a time serializes latency that doesn't need to be serial.

That anatomy hands you four levers, roughly in order of leverage: stop re-reading what you already summarized (incremental summarization), trim what's left (dedupe and truncate), use a cheaper model where quality holds (right-size), and stop waiting on calls one at a time (concurrency).

Lever 1: Incremental Summarization — Stop Rereading the Whole Log

A ticket log is not static. A busy incident ticket accumulates thousands of log lines over its life, and every status update needs a fresh summary. The naive approach re-sends the entire log to the model on every regeneration, so the same thousands of lines are paid for again and again, and the token bill grows with the ticket.

Instead, we cache each ticket's latest summary in Redis and treat it as running state. When the ticket updates, we send the model the cached prior summary plus only the new log lines since we last ran, and ask it to fold that delta into an updated summary. The model reads a short summary and a small delta rather than the full history — and because logs can run to thousands of lines, this was where the biggest token savings came from.

The guardrails: we periodically force a full re-summarization so small errors don't compound across many incremental updates, and the cached summary is always refreshed against new log lines rather than served stale. This is the classic rolling summarization pattern — cheap to run, and it turns an ever-growing log into a bounded, near-constant cost per update.

Incremental summarization: instead of re-sending the entire growing log on every update, we send the cached summary plus only the new lines - so cost stays flat as the ticket grows.

Lever 2: Token and Context Trimming — Make Each Call Smaller

Incremental summarization stops us re-sending the whole history; trimming shrinks what's left on the calls we do make. Because input tokens dominate a summarizer's cost, we squeezed the rest:

• Deduplicate the log: Machine-generated logs repeat the same alarm and heartbeat lines over and over. Collapsing those duplicates removed large redundant blocks before they ever hit the model.
• Truncate to what matters: Instead of shipping a ticket's entire history, we cap the window to the most recent and most relevant events.
• Cap the retrieved context: More retrieved chunks from the vector store is not automatically better -  each one is pure input cost. We limited how many we pull.
• Cap max output tokens: Summaries are short by design, so a hard output ceiling prevents runaway generations and, because generation time tracks output length, directly trims latency.

Together, these cut a large share of the remaining input on every call. Fewer tokens in and out is the rare optimization that reduces cost and latency at the same time.

Lever 3: Model Right-Sizing — Pay for Quality Only Where It Matters

Not every ticket deserves the flagship model. We select the model per ticket from business signals — severity, number of customers impacted, and revenue impact — so premium-model tokens go to the tickets that actually warrant them (major outages, high-value customers) while routine tickets get a smaller, cheaper, and faster model. The lowest-severity tickets skip the LLM altogether. A non-LLM extractive summary (key fields, error counts, first and last events) costs zero tokens. It's the same severity tiering we use to keep the pipeline resilient — here it doubles as the cheapest tier of all.

The guardrail is a quality bar: before trusting a cheaper tier for a class of tickets, we validate its summaries against the flagship's, and when the signal is ambiguous, we route up, not down. Right-sizing is a double win: smaller models cost less and respond faster, so it reduces both spend and typical response time.

Lever 4: Parallel Execution — Stop Waiting in Line

Summaries are independent of one another, so processing them one at a time was leaving latency on the table. Running calls concurrently (async) collapses the wall-clock time to clear a burst of tickets and sharply improves perceived latency when several land at once.

The one caveat is that unbounded concurrency will happily blow past your provider's tokens-per-minute limit. So concurrency runs under a shared token budget: as many calls in flight as the budget allows, and no more. You get the throughput without tripping the ceiling.

Results

No single trick did it; the levers compounded. Together they're bringing our blended LLM cost down by roughly 60% so far, while making summaries faster and steadier under load:

The biggest surprise wasn't an exotic trick — it was simply not paying to re-read tokens we had already summarized. The other three levers compounded on top.

What We Watch For

• Summary drift: Because each update builds on the previous summary rather than the raw log, a small error can propagate across updates. A periodic full re-summarization from the original log resets any accumulated drift.
• Quality regressions: Over-truncation can drop the one log line that mattered, and a too-cheap model can miss nuance. We keep a periodic eval/spot-check, so cost cuts don't quietly erode quality.
• Don't optimize noise: Measure first; a low-volume path rarely repays the complexity.

Takeaways

• Measure where the tokens go – For a summarizer, input dominates - optimize that first.
• Biggest wins first – Reuse prior summaries instead of re-reading the whole log, trim what you do send, and right-size the model. Exotic tricks come later.
• Concurrency is a latency lever, not a cost lever – and it must respect your rate limits.
• Guard quality whenever you trim or downshift – pair every cost cut with an eval.