Why LLM Pipelines Fail in Production and How Temporal and Kafka Fix Them

A production LLM pipeline is rarely just a prompt and a response. It typically combines retrieval, prompt rendering, model inference, output shaping, validation, persistence, and downstream actions. That broader shape is why many systems look stable in a demo and then become fragile under live traffic. The model call is only one component; the operational problem is the workflow around it. Provider APIs impose rate limits, structured outputs still need application-level checks, and external calls introduce failure ambiguity that ordinary request-response code does not handle well.

Where the Breakage Starts

Most production failures happen between steps, not inside the prompt. A request enters an API, context is loaded, a model call is sent, the response is parsed, a downstream action is triggered, and a record is written. If the provider generated output but the network dropped before the caller saw it, the system no longer has a clean answer to whether the operation should be retried or treated as complete. Kafka’s default delivery model is at least once, and Temporal’s documentation is explicit that activities may be retried and therefore should be idempotent. That combination makes duplicate side effects the default risk unless the pipeline is designed around durable state and idempotent writes. 

Duration creates the second breakage pattern. Ingestion may fan out across thousands of chunks, while a risky action may need approval hours later. Temporal workflows can receive external write events through Signals, and durable timers persist across worker and service downtime, so a workflow can pause without collapsing into callback code and scheduled cleanups. Temporal also requires workflow logic to remain deterministic during replay and provides versioning methods so new executions can adopt new code while long-running executions remain on compatible paths. Those concerns are not edge cases in LLM systems; they are normal once the pipeline extends beyond a single synchronous call. 

Output shape is another common source of confusion. OpenAI’s Structured Outputs guide exists because unconstrained text is not a reliable contract; the feature is designed to enforce a supplied JSON Schema and avoid missing required keys or invalid enum values. But schema compliance is only the first gate. A response can be structurally valid and still be semantically wrong, stale, or unsafe to automate. Production failures happen when formatting success is mistaken for business correctness.

Why Kafka solves only part of it

Kafka is a strong fit at the ingestion boundary because it turns synchronous pressure into a durable stream of work. Kafka topics are partitioned, ordering is guaranteed within a partition, and each partition is consumed by exactly one consumer in a consumer group at a given time. Consumers also control offsets and can rewind to replay records. That combination is well suited to bursty LLM demand, key-based ordering, and reprocessing after a model or prompt change.

public void submitRequest(LlmRequest request) {
    LlmRequestEvent event = new LlmRequestEvent(
        request.requestId(),
        request.tenantId(),
        request.documentId(),
        request.templateId()
    );

    kafkaTemplate.send("llm.requests", request.requestId(), event);
}

This pattern keeps the API narrow. The service records intent by publishing an event keyed by requestId; Kafka’s default partitioning uses the key hash, so related records land on the same partition and preserve that partition’s order. Kafka’s producer is also optimized for batching, and its pull-based consumer model lets downstream services fall behind and catch up instead of being overwhelmed by broker-driven push traffic. That is useful when inference latency varies, and demand arrives in bursts. 

But Kafka only states that work was published and later consumed. It does not know whether retrieval already succeeded, whether a model provider timed out after actually producing output, or whether persistence ran before a crash. Offsets capture consumption position, not business completion. Kafka is excellent for transport, buffering, replay, and fan-out, but insufficient as the sole control plane for a multi-step inference process. 

Why Temporal Changes the Outcome

Temporal addresses the state problem directly. Its model is durable execution: workflows advance through an event history stored by the Temporal service, and that history is what allows an execution to recover from a crash and continue making progress. Worker crashes, network interruptions, and infrastructure outages are handled differently from ordinary application failures because the workflow state is not reconstructed from logs after the fact; it is already part of the execution record.

@KafkaListener(topics = "llm.requests")
public void onRequest(LlmRequestEvent event, Acknowledgment ack) {
    InferenceWorkflow workflow = workflowClient.newWorkflowStub(
        InferenceWorkflow.class,
        WorkflowOptions.newBuilder()
            .setWorkflowId(event.requestId())
            .setTaskQueue("llm-inference")
            .build()
    );

    try {
        WorkflowClient.start(workflow::run, event);
    } catch (WorkflowExecutionAlreadyStarted ex) {
        log.info("Workflow already started for {}", event.requestId());
    }

    ack.acknowledge();
}

The important detail is the workflow identifier. Temporal guarantees workflow ID uniqueness within a namespace and prevents another open workflow with the same ID from starting, which turns duplicate Kafka deliveries into a safe re-entry case instead of parallel duplicate execution. The Kafka listener acknowledges the record after the workflow start is accepted, not after the entire inference path finishes. Kafka remains the transport layer; Temporal becomes the durable execution layer for the request. 

Inside that workflow, each external operation should sit in an activity with explicit timeout and retry policy rather than inside scattered retry loops. Temporal’s retry model is declarative; activities retry by default, and the platform documentation recommends making activities idempotent and granular because a retry re-executes the whole activity. That fits LLM systems unusually well, since retrieval, prompt rendering, model invocation, validation, and persistence rarely fail for the same reason.

private final LlmActivities activities = Workflow.newActivityStub(
    LlmActivities.class,
    ActivityOptions.newBuilder()
        .setStartToCloseTimeout(Duration.ofSeconds(45))
        .setRetryOptions(RetryOptions.newBuilder()
            .setMaximumAttempts(5)
            .setInitialInterval(Duration.ofSeconds(2))
            .build())
        .build()
);

public InferenceResult run(LlmRequestEvent event) {
    Context context = activities.loadContext(event.documentId());
    Prompt prompt = activities.renderPrompt(event.templateId(), context);
    ModelResponse response = activities.callModel(prompt, event.requestId());
    return activities.validateAndPersist(event.requestId(), response);
}

That separation also makes the final validation step honest. Structured output may guarantee a schema, but business rules still need to decide whether the result is usable, and persistence still needs an idempotent write keyed by requestId so a retry cannot create duplicate approvals, tickets, or rows. In practice, this is the difference between a pipeline that merely retries and a pipeline that resumes safely. 

Why the Handoff Works in Production

The pattern that holds up is a clean handoff. Kafka should own ingress, buffering, replay, and downstream fan-out. Temporal should own the lifecycle of one logical request. When a workflow completes, it can publish a result event for indexing, notifications, analytics, or billing, and each downstream system can consume that event independently. Kafka moves facts through the platform. Temporal ensures the process that produced those facts reaches a correct terminal state. 

This split also improves the cases that usually appear after launch. Human approval can arrive as a Temporal Signal without keeping compute hot. A workflow can pause for days using a durable timer and continue after downtime. Workflow code can be versioned so new executions take a new path while long-running executions stay on compatible logic, which is important because replay depends on deterministic workflow behavior. 

The reason LLM pipelines fail in production is not that language models are impossible to operationalize. The reason is that they are often deployed as request handlers when they are actually distributed workflows with uncertain latency, replay risk, and side effects. Kafka fixes the transport problem by providing durable, replayable streams with partition ordering and load decoupling. Temporal fixes the execution problem by persisting progress, surviving worker failure, and making retries, timeouts, and human pauses part of the design instead of a patch applied after incidents. When those two responsibilities are separated cleanly, an LLM pipeline stops behaving like an experimental chain of API calls and starts behaving like a production system.