Batching Strategies: Throughput vs Latency in Practice

Production inference is fundamentally a scheduling problem. Once you've addressed serving architecture and memory, the next question isn’t whether the GPU can do the work; it's how to schedule it.

That decision begins with understanding the workload: how requests arrive, how long they run, which are latency-sensitive, and which can tolerate delay.

This is where many production deployments encounter their next engineering challenge. Serving architecture establishes how an inference system is organized. Memory management determines how efficiently it uses GPU memory. Batching answers a different question:

How do you keep expensive GPUs doing useful work while still meeting latency service-level objectives (SLOs)?

Production workloads introduce competing objectives that don't exist during experimentation. Production LLM systems serve requests that arrive unpredictably, vary dramatically in length, and compete for the same hardware resources. The engineering challenge isn’t batching requests together; it's deciding how work should be scheduled as those requests enter and leave the system.

Production Traffic Is Inherently Uneven

Most proof-of-concept environments present an idealized view of inference. Requests arrive at predictable intervals. Prompt lengths are similar. Response lengths are relatively consistent.

Production looks nothing like that. Prompt lengths, output lengths, arrival patterns, and latency expectations differ. But production schedulers have to account for all of them simultaneously. That heterogeneity is what transforms batching from a simple grouping exercise into a scheduling problem.

One user may ask a simple question that requires only a few tokens. Another may submit a lengthy document that requires a large context window and an extended response. Some requests are interactive and latency-sensitive. Others are background jobs that can tolerate additional delay. Treating all of those requests as if they were identical creates unnecessary inefficiencies. That is why batching strategies that work well during experimentation often become liabilities in production.

Static Batching Breaks Under Real Workloads

Static batching is straightforward: Collect a group of requests, execute them together, and wait until the batch finishes before beginning the next one. That approach works well for many traditional AI workloads because inputs generally require a similar amount of computation.

Autoregressive generation behaves differently. Each request generates tokens at its own pace. Some complete almost immediately. Others continue generating for hundreds or even thousands of decoding iterations. The result is that the shortest requests spend much of their time waiting for the longest one to finish.

This is the convoy effect. The GPU is effectively synchronized to the slowest request in the batch. The GPU is occupied, but completed or short-running requests cannot release their resources until the longest request in the batch finishes.

As production workloads become more diverse, those idle cycles accumulate. Throughput falls, latency increases, and expensive GPU resources spend more time waiting than performing useful work.

Bigger Batches Are Not Always Better

The natural response is often to increase batch size. Larger batches generally improve GPU utilization, but they also introduce a tradeoff. Every request must wait until enough additional work arrives to fill the batch, increasing queueing delay before execution even begins.

Once execution starts, larger batches can also increase Time Per Output Token (TPOT). A configuration that delivers acceptable TPOT at one batch size may exceed the application's latency SLO as the batch grows. Higher GPU utilization provides little benefit if response times no longer meet application requirements.

This is the central balancing act of production inference. Higher throughput often comes at the cost of higher latency. The engineering challenge is determining where that operating point belongs.

An internal document-processing pipeline may prioritize maximum throughput because latency is relatively unimportant. A customer-facing assistant may willingly sacrifice some throughput to preserve fast responses. Neither choice is universally correct. The workload determines the objective. The batching strategy should support it.

Continuous Batching Changes the Scheduling Model

Modern serving systems approach the problem differently. Rather than waiting for an entire batch to complete, continuous batching schedules work at each decode iteration. Finished requests leave immediately. New requests enter immediately. The scheduler is no longer managing batches. It is managing decoding iterations.

Requests enter the batch as capacity becomes available. Completed requests leave immediately. New requests replace them without waiting for the entire batch to finish. This seemingly simple change fundamentally alters GPU utilization.

Instead of waiting for the longest request to complete, the scheduler continuously replenishes the batch with new work. GPU utilization becomes constrained primarily by queue depth rather than by the slowest request in the system. The scheduler is no longer waiting for an entire batch to complete. It's continuously looking for the next unit of useful work. 

The objective shifts from building larger batches to keeping the GPU continuously productive.

Long Prompts Introduce Another Challenge

The prefill phase processes the entire prompt in a single forward pass, while the decode phase generates one token at a time. These phases place very different demands on the hardware. Prefill is compute-bound. Decode is memory-bandwidth-bound. Long prompts can monopolize GPU compute, delaying interactive requests that are already in the decode phase.

Chunked prefill addresses this by breaking large prompts into smaller segments that can be interleaved with ongoing decode operations. Rather than allowing one long prefill operation to dominate the GPU, the resource scheduler alternates between prompt processing and token generation, allowing engineers to balance Time to First Token (TTFT) for new requests against Time Per Output Token (TPOT) for active ones.

The result is a more balanced system.

Time to First Token (TTFT) for new requests remains reasonable while Time Per Output Token (TPOT) for active requests stays within acceptable service-level objectives. Once again, the objective is balance — not maximization.

Scheduling Becomes Policy

As production systems mature, batching becomes less about grouping requests and more about deciding which work should execute next.

• Interactive applications
• Background batch jobs
• High-priority users
• Low-priority workloads

Each may require different treatment. Some workloads don't require interactive latency at all. If a business process can tolerate scheduled execution, maximizing throughput may be the better engineering decision than minimizing response time.

Modern serving systems increasingly make resource scheduling decisions at the token level rather than at the request level. GPU time becomes a shared resource allocated according to policy instead of simple arrival order.

That policy also extends to admission control. Available GPU capacity is not the same thing as available latency headroom.

Accepting every incoming request may maximize utilization for a short period, but it can also push latency beyond acceptable SLOs for every user in the system. Well-designed serving systems recognize that preserving predictable latency is often more valuable than accepting one additional request. In production, saying "not yet" is frequently better than saying "yes" to everyone.

Throughput and Latency Are Negotiated — Not Maximized

One theme runs consistently through every batching strategy. There is no universally optimal configuration. The appropriate batch size, scheduling policy, and admission thresholds all depend on the workload and the service-level objectives the application must satisfy. There is no universally optimal configuration because every production deployment is balancing a different set of constraints.

A nightly batch pipeline should not be engineered like an interactive coding assistant. Likewise, a customer support chatbot should not inherit the scheduling strategy of an offline document-processing system simply because both happen to use the same foundation model.

The model may remain unchanged. The serving behavior should not. The objective isn't maximum throughput. It's the highest sustainable throughput that still satisfies the application's latency SLOs.

Looking Ahead

Serving architecture organizes the inference system. Memory management ensures efficient GPU memory use. Batching strategies determine how available resources are converted into useful work. 

The final layer focuses on extracting even more performance from the same hardware. In the next article, we'll examine inference optimization techniques – including quantization, speculative decoding, FlashAttention, and distillation – and how carefully sequencing those optimizations can dramatically reduce cost and latency without changing the underlying model.