The Pipeline Failed, But the Infrastructure Changed

Most infrastructure pipelines reduce an operation to two facts: the command that ran and its exit code.

That is enough to tell whether the process completed. It is not enough to tell what happened to the infrastructure.

A deployment can create a network and start a virtual machine before failing during configuration. The pipeline reports failure while the provider reports live resources. Any runtime coordinating these calls needs a state model for that gap.

Better logs help explain the error. They do not solve the state problem.

The Boolean Result Is the Wrong Data Model

The first design mistake is treating infrastructure execution as a boolean transaction. A run is not simply successful or rolled back.

The process may stop after a remote system accepts a change. A timeout can leave the caller uncertain whether the request failed or only its response was lost. Retrying may duplicate work or collide with the existing object.

DZone's article on practical transaction handling in microservices describes the same failure boundary across remote services. One operation can succeed while the process responsible for recording it fails. A later retry then needs idempotency and durable transaction identity, not optimism.

Infrastructure workflows have the same distributed-systems property.

Separate Three Kinds of State

A useful implementation starts by separating three facts that are often mixed together:

The process result belongs to the current invocation. Provider state belongs to the systems that own the resources. Operation state connects the two across runs.

In Your Automation Pipeline Is Not a Source of Truth, I argued that pipeline logic should execute declared intent rather than become the authority for it. Partial failure exposes the matching runtime boundary: the pipeline result cannot stand in for the current state of remote resources.

This does not replace infrastructure-engine state. HashiCorp defines Terraform state as the mapping between configured resource instances and real-world remote objects, along with dependency and provider metadata. An orchestration runtime needs a complementary record of the operation, stage, inputs, outputs, and valid recovery actions.

DZone's Terraform debugging guide notes that discrepancies between planned and actual state are a central source of deployment confusion. A runtime should not add another discrepancy by retaining a terminal marker from an earlier run after a new mutation begins.

Persist Mutation Intent Before Calling the Provider

The critical transition is the point immediately before provider execution.

Validation and preflight can fail without changing remote resources. Once they pass, the runtime should persist a nonterminal state before making a mutating call. If it cannot commit that marker, it should refuse to invoke the provider.

A simplified execution path looks like this:

preflight(target, inputs)

write_state(
    target=target,
    status="running",
    active_command="apply",
    resolved_inputs=inputs,
    run_id=run_id,
    required=True,
)

try:
    result = provider.apply(target, inputs)
except Exception as exc:
    write_error_state(target, run_id, exc)
    raise

if result.ok:
    write_success_state(target, run_id, result.outputs)
else:
    write_error_state(target, run_id, result.error)

The required=True behavior is more important than the syntax. A runtime that cannot record the start of a mutation should not create remote state that the next run cannot explain.

The process can stop after writing running but before the provider call. That leaves conservative uncertainty even when nothing changed remotely. The reverse error is more dangerous: keeping an old destroyed marker after a provider may have created a resource.

Failure Must Preserve Recovery Context

An exception handler should not replace the operation record with an empty failure object. It should update the record while retaining facts that remain useful.

At minimum, keep:

• target and state-instance identity;
• run identifier and timestamp;
• active or failed command;
• concise failure detail;
• last-known provider outputs;
• resolved input location;
• sanitized rerun input location; and
• the run-record path.

A retained resource identifier may be stale. Resolved inputs explain what the runtime attempted, but they do not prove that the provider accepted it. The provider remains authoritative for current existence and attributes.

The difficult first-run case should be explicit. A provider may create an object and then fail before returning its identifier. No state model can preserve an output it never received. The operation record still narrows recovery by preserving the request identity, target, inputs, time, and failure point.

Dependency Status Must Remain Truthful

Blueprints and deployment graphs add another failure mode: status propagation.

Assume the graph contains four ordered steps:

network foundation  ok
compute instance    error
host bootstrap      pending
service probe       pending

Only the virtual-machine step failed. Configuration and health verification did not run. Marking all three as failed loses the execution boundary and makes resume logic harder to reason about.

A practical status model needs at least:

• ok for completed work;
• error for attempted work that did not complete;
• pending for work blocked by a dependency;
• retained for resources deliberately kept; and
• destroyed or absent for terminal cleanup state.

The distinction determines whether a rerun should inspect, resume, skip, compensate, or destroy each node.

Retry, Reconcile, Compensate, or Destroy

Recovery should select an action from state rather than apply one universal retry rule.

Retry suits a transient failure when the operation is idempotent or protected by a stable request identity. DZone's discussion of failure in long-running processes uses the classic uncertain-response example: the remote system may have processed a request even though the caller timed out.

Reconciliation suits uncertain state. Query the provider using known identity and inputs, then update the operation record from current facts.

Compensation suits completed work that must be reversed through domain-specific actions. Microsoft's Compensating Transaction pattern emphasizes that compensation is itself stateful, resumable work. It may not run in exact reverse order, and it may fail.

Google Cloud Workflows similarly recommends retries and the saga pattern for multi-service operations. A failed step can trigger compensating transactions for earlier successful steps, but only when the workflow has retained enough information to perform them.

Destroy is appropriate when the intended outcome is absence. It should operate from error state as well as successful state because a failed apply may have left resources behind.

A Delete Request Is Not Terminal State

Teardown needs the same discipline as creation.

Destroy logic should skip only resources already confirmed absent or destroyed. An error marker is potentially live and remains eligible for provider inspection and removal.

Parent-owned resources need another guard. Removing a virtual machine may also remove guest-level state that cannot be reached independently afterward. The child can be marked destroyed only after the parent removal succeeds. If the parent remains, the child remains unresolved.

Kubernetes finalizers offer a useful comparison. A delete request adds a deletion timestamp, but the object remains in a terminating state until required cleanup completes. The request is accepted before deletion is complete.

Infrastructure runtimes should make the same distinction between requested, in-progress, and verified absence.

Test the Contract, Not Just the Happy Path

The failure contract should be expressed through regression tests. Useful cases include:

1. An old destroyed marker becomes running before provider execution.
2. The provider is not called when the required state write fails.
3. A failed initial apply creates nonterminal error state.
4. Earlier outputs survive a later failed mutation.
5. A successful apply replaces running with ok.
6. Preflight failure leaves prior terminal state unchanged because mutation did not begin.
7. An error state is included in destroy.
8. A child is marked destroyed only after its owning parent is confirmed gone.

The key assertion is not only which return code was produced. It is what durable state exists before, during, and after the simulated failure.

One Implementation Example

HybridOps Core, the open-source runtime I maintain, implements this contract after testing stale state across partial provider mutations. The runtime writes running before apply, deploy, or import; blocks provider execution if that write fails; preserves prior outputs on error; and keeps error state eligible for blueprint destroy.

The implementation and regression tests are public. Related changes also separated pending steps from failed steps and made parent-owned teardown conditional on successful parent removal.

The important point is not the project-specific field names. It is the boundary they enforce: remote mutation cannot begin under an operation state that still claims the resource is absent.

Durable Failure Is Better Than False Success

Infrastructure automation cannot make several independent provider calls atomic simply by placing them behind one command. It can make partial failure recoverable.

That requires a nonterminal marker before mutation, retained recovery context, truthful dependency status, and teardown that ends only after absence is verified.

The pipeline result still matters. It tells the team that the requested operation did not finish. Durable state tells them what to do next.