Orchestrating Small Language Models Without Losing Events or Context

Reliable orchestration for small language models depends less on model sophistication than on the durability of event flow and state. Under the assumptions used here — small model instances, little or no local state, Kafka as the event backbone, Temporal as the orchestration layer and durable state store, and Java as the runtime — the safest design is to treat model invocations as replayable side effects, Kafka as the transport and ordering substrate, and Temporal Workflow state as the canonical record of conversational progress. 

In that design, Kafka provides high-throughput append-only event delivery and partition-local ordering, while Temporal persists Workflow Event History and can replay execution after failures. Exactly-once semantics remain meaningful inside Kafka’s consume-transform-produce boundary when transactions and read_committed are used, but once processing crosses into external systems such as model APIs, durable activities, or databases, correctness comes from idempotency, deduplication, sequence checks, and reconciliation rather than from a global exactly-once guarantee. 

Assumptions

The most productive baseline is a narrow one. Each conversation, task, or model session is keyed so related events land on the same Kafka partition, preserving order only where order actually exists: within one partition, not across the topic. 

Each workflow instance owns one conversational state machine, stores the minimal context needed to decide the next action, and invokes model calls through Temporal Activities so failures, retries, and timeouts are visible and durable. Large prompts, attachments, or long transcripts are not kept as incidental JVM memory because Temporal persists inputs and outputs in Event History and large histories degrade replay latency; those artifacts belong in external storage with durable references held in workflow state. 

Analysis

The central engineering mistake in LLM orchestration is to confuse transport delivery with business completion. Kafka can guarantee at-least-once delivery by processing records before committing consumer offsets, and it can provide exactly-once behavior for Kafka-to-Kafka pipelines by atomically updating produced records and consumed offsets with transactions. Kafka’s own design documentation is explicit that the producer is the transactional component and that read_committed is advisable when aiming for exactly-once processing. 

The same documentation also makes clear why the guarantee weakens at system boundaries: once consumed data must be coordinated with an external state store or side effect, the problem becomes cross-system consistency rather than log delivery. In a Temporal-based model pipeline, that means Kafka should usually be treated as the durable ingress path, while Temporal owns the authoritative notion of whether an event was applied to a conversation state machine. 

That separation suggests a simple rule. Offsets are transport progress; workflow state is semantic progress. A consumer should therefore commit offsets only after handoff to a durable semantic owner. In this architecture, that owner is the Temporal workflow receiving a signal. Temporal workflows behave like stateful services that receive Signals, Queries, and Updates, and the platform persists Event History so a crashed worker can replay the workflow and resume from the last recorded event. Signal handlers are allowed to mutate workflow state, and blocking coordination can be expressed safely with Workflow.await. Activity retries are configured through ActivityOptions and RetryOptions, with heartbeat support for long-running calls. 

@WorkflowInterface
interface ModelFlow {
    @WorkflowMethod void run(String sessionId);
    @SignalMethod void onEvent(ModelEvent event);
    @QueryMethod long lastAppliedSequence();
}

private final ModelActivities activities =
    Workflow.newActivityStub(
        ModelActivities.class,
        ActivityOptions.newBuilder()
            .setStartToCloseTimeout(Duration.ofSeconds(20))
            .setRetryOptions(
                RetryOptions.newBuilder()
                    .setInitialInterval(Duration.ofMillis(250))
                    .setMaximumAttempts(5)
                    .build())
            .build());

private final NavigableMap<Long, ModelEvent> pending = new TreeMap<>();
private long nextSequence = 1;
private ConversationState state = ConversationState.empty();

@Override
public void onEvent(ModelEvent event) {
    pending.putIfAbsent(event.sequence(), event);
}

@Override
public void run(String sessionId) {
    for (;;) {
        Workflow.await(() -> pending.containsKey(nextSequence) || state.closed());
        if (state.closed()) break;
        var event = pending.remove(nextSequence);
        state = activities.applyEvent(state, event);
        nextSequence = event.sequence() + 1;
    }
    Workflow.await(Workflow::isEveryHandlerFinished);
}

@Override
public long lastAppliedSequence() {
    return nextSequence - 1;
}

This workflow fragment does three important things at once. The @SignalMethod declares asynchronous event ingress, the @QueryMethod exposes durable progress for reconciliation, and the activity stub attaches retry policy directly to the state transition that may call a model endpoint or another dependency. The pending map is not a queue for throughput; it is a reordering guard. If Kafka redelivers a message or an upstream retry arrives out of sequence, putIfAbsent and the nextSequence gate prevent semantic duplication and preserve per-session causality. Finishing the run only after Workflow.isEveryHandlerFinished() avoids the Temporal-documented failure mode where a workflow completes or continues-as-new while a handler is still waiting on asynchronous work. 

The matching Kafka consumer must be deliberately conservative. Automatic commits are inappropriate because they advance transport progress in the background regardless of semantic application. Manual synchronous commits make the boundary explicit, and Kafka documents that committed offsets are the secure restart position, and that commitSync should write the next offset, meaning lastProcessedOffset + 1. The consumer is also not thread-safe, so per-partition in-order handling is easiest when one poll loop owns one consumer instance and performs durable handoff before commit.

void pollLoop() {
    consumer.subscribe(List.of("model-events"));
    while (running.get()) {
        var records = consumer.poll(Duration.ofSeconds(1));
        for (var partition : records.partitions()) {
            var batch = records.records(partition);
            for (var record : batch) {
                var eventId = header(record, "event-id");
                if (!inbox.tryInsert(eventId, record.topic(), record.partition(), record.offset())) {
                    continue;
                }
                var workflow = client.newWorkflowStub(ModelFlow.class, record.key());
                workflow.onEvent(ModelEvent.from(record));
                inbox.markApplied(eventId);
            }
            var nextOffset = batch.get(batch.size() - 1).offset() + 1;
            consumer.commitSync(Map.of(partition, new OffsetAndMetadata(nextOffset)));
        }
        if (inbox.backlog() > 50_000) consumer.pause(consumer.assignment());
        else consumer.resume(consumer.assignment());
    }
}

The durable inbox is the effective-once bridge. If the process crashes after signaling Temporal but before committing offsets, Kafka may redeliver, yet tryInsert suppresses reapplication. If upstream producers use Kafka transactions, the consumer should read with isolation.level=read_committed so aborted records stay invisible; Kafka’s configuration reference notes that read_committed returns only committed transactional messages and withholds records past the last stable offset while open transactions exist. Backpressure also belongs here. Kafka exposes pause and resume without forcing a group rebalance, and monitoring guidance explicitly recommends watching lag, fetch rate, poll timing, and commit latency to ensure consumers are keeping up. 

Context propagation is easiest when context is split into stable metadata and mutable conversational state. Stable identifiers such as trace ID, tenant, policy version, and conversation key belong in Kafka headers and Temporal headers so they survive hops across services and activities; Kafka’s ProducerRecord supports headers, and Temporal context propagators move custom key-value data across workflow, activity, and child-workflow boundaries. Mutable context, by contrast, should not live in worker memory or ad hoc caches. It belongs in the workflow state, often as a compact summary plus references to offloaded artifacts. Temporal’s documentation explicitly warns that all activity inputs and outputs are persisted, that long AI-style conversations grow history, and that large histories degrade workflow-task latency. For long-running sessions, Continue-As-New provides a checkpoint boundary with a fresh Event History while preserving the workflow identity chain. 

Reconciliation closes the last reliability gap. Even with careful commits, outages, manual replays, or producer bugs can create suspicion that a workflow missed an event. Temporal queries are read-only and must not mutate state or block, which makes them ideal for asking a workflow for its durable high-water mark and replaying any gap from the event store.

void reconcile(String workflowId, long durableHighWatermark) {
    var workflow = client.newWorkflowStub(ModelFlow.class, workflowId);
    long applied = workflow.lastAppliedSequence();
    eventStore.readRange(workflowId, applied + 1, durableHighWatermark)
              .forEach(workflow::onEvent);
}

This pattern works because the workflow does not trust delivery history alone; it trusts its own durable state. Observability then becomes the enforcement layer for those guarantees. Kafka should surface lag, request latency, retry rates, poll gaps, and buffer exhaustion, while Temporal should emit metrics through Micrometer, trace activity and workflow paths, and expose searchable workflow metadata through Search Attributes. Temporal also recommends monitoring replay latency because large histories, payload sizes, and cache churn drive recovery cost. Together, these signals reveal the difference between a system that is slow, a system that is duplicating work, and a system that is actually losing context. 

Conclusion

Orchestrating small language models without losing events or context is fundamentally a durability problem, not a prompt-engineering problem. Kafka should be used for ordered transport and scalable ingestion, but semantic completion should be anchored in Temporal’s durable workflow state, where signals, sequence gates, retryable activities, queries, and replay make failures recoverable rather than ambiguous. Exactly-once remains valuable inside Kafka’s transactional envelope, yet end-to-end correctness across model calls and other side effects comes from explicit idempotency, deduplication, reconciliation, and bounded context management with external storage and continue-as-new. 

In a Java stack, that combination yields an architecture where duplicates become harmless, ordering becomes explicit, back pressure becomes controlled, and context survives crashes because it is recorded in durable state instead of being left in process memory.