Commissioning at Scale Is a Sequencing Problem, Not a Testing Problem

The first site I ever failed to place in service passed every acceptance test I wrote for it. Cameras streamed. Switches held their uplinks under a simulated fiber cut. Audio was intelligible at every measurement point. The paperwork was clean.

It still sat dark for three weeks, because the room where one of the redundant paths terminated belonged to a different crew on a different contract with a different completion date, and nobody had drawn that edge on any schedule. My validation was fine. My ordering was wrong.

I have spent about a decade deploying networked systems across large numbers of physical sites in a public transit environment, running in parallel, under live operating conditions, with no maintenance window that lets you take the whole thing down. Surveillance endpoints, station-level LANs, public address, intercom, two-way emergency communications. The engineering content of any single site is not especially exotic. What breaks at scale is not the depth of your testing. It is the order in which work becomes possible.

Software teams hit the identical wall the first time they go from one deployment target to sixty. So I want to lay out the methodology that actually held up, in terms that map onto a release pipeline, because that is the shape of the problem.

Sites Are Environments, Not Projects

The instinct on a multi-site program is to treat each site as a self-contained project: design it, build it, test it, close it, move to the next. It feels rigorous. It is also the single most expensive decision available to you, because it forces every skill in the program to be present at every site, sequentially, and your throughput collapses to the speed of your scarcest crew.

The reframe that fixed it for me: a site is an environment, not a project. Environments are provisioned from a shared definition. They differ by a small set of declared parameters and nothing else. If two sites differ in a way that is not captured in that parameter set, that difference is a defect in my design, not a fact about the world.

Once I held that line, everything downstream got cheaper. Configuration became generation. Testing became parameterization. And the schedule became a dependency graph instead of a list.

Addressing Is a Schema, Not a Spreadsheet

For the first phase, I inherited what most programs have, which is a spreadsheet of IP assignments maintained by whoever touched it last. It works for a dozen devices. At a few hundred, it starts producing collisions and orphaned addresses, and at a thousand it produces the worst failure mode there is: a device that answers on the network but is not the device you think it is.

So the addressing plan became a schema with a deterministic derivation, and the spreadsheet became a generated artifact rather than a source of truth.

# sites/site-042.yaml
site_id: 042
tier: modernization
mgmt_supernet: 10.42.0.0/16

subnets:
  management:   { offset: 0,  size: 24 }
  surveillance: { offset: 16, size: 22 }
  audio:         { offset: 32, size: 24 }
  intercom:      { offset: 40, size: 24 }

endpoints:
  surveillance:
    - { tag: CAM-P-01, zone: platform_north, switch: SW-A, port: 1, poe: true }
    - { tag: CAM-P-02, zone: platform_south, switch: SW-A, port: 2, poe: true }
  intercom:
    - { tag: INT-EL-01, zone: elevator_lobby, switch: SW-B, port: 7, critical: true }

Addresses are derived from site_id, the subnet offset, and the endpoint's index within its class. Nobody assigns an address by hand. A rendering step turns this into switch configuration, into the label schedule the field crew prints, and into the test suite. One input, three outputs that cannot drift from each other.

def render_switch_config(site, switch_id):
    lines = [f"hostname {site['site_id']}-{switch_id}"]
    for cls, eps in site["endpoints"].items():
        vlan = VLAN_MAP[cls]
        for i, ep in enumerate(e for e in eps if e["switch"] == switch_id):
            lines += [
                f"interface Gi1/0/{ep['port']}",
                f" description {ep['tag']} {ep['zone']}",
                f" switchport access vlan {vlan}",
                " spanning-tree portfast" if not ep.get("critical") else "",
                f" power inline {'auto' if ep.get('poe') else 'never'}",
            ]
    return "\n".join(l for l in lines if l)

The value here is not elegance. It is that a configuration error is now a class of error rather than an instance of one. When I found a wrong VLAN on one intercom port, I knew immediately whether it was a typo at one site or a bug that had shipped to forty. That distinction is the difference between an afternoon and a month.

Batch by Equipment Type, Not By Site

Here is the sequencing change that bought back the most schedule.

Crews are specialized. The people who terminate and splice fiber are not the people who mount and aim cameras, who are not the people who tune audio for intelligibility, who are not the people who witness a formal acceptance test with an inspector present. If you sequence site by site, each of those crews shows up, works for a day or two, and leaves, and you pay the mobilization cost every time. Worse, the sequence is serial per site, so the whole program moves at the pace of one site's critical path multiplied by the number of sites.

Group by equipment class across sites instead. The fiber crew runs its scope across a cluster of sites in one pass. The endpoint installers follow a fixed number of sites behind. The commissioning engineer follows them. It looks exactly like a staged pipeline, and it behaves like one: the throughput is set by the slowest stage, and work in progress between stages is inventory you are carrying.

The thing that makes this legal, rather than reckless, is that batching only works when the stage boundary is a real gate with a machine-checkable entry condition. Otherwise, you are just moving unfinished work forward and discovering it later, at the most expensive possible moment, which is with an inspector standing next to you.

GATES = {
  "physical_ready": lambda s: s.fiber_certified and s.power_energized,
  "network_ready":  lambda s: s.config_pushed and s.uplinks_redundant,
  "endpoint_ready": lambda s: s.all_tags_resolve and s.poe_budget_ok,
}

def promotable(site, stage):
    return all(check(site) for name, check in GATES.items()
               if STAGE_ORDER.index(name) <= STAGE_ORDER.index(stage))

A site that fails physical_ready does not get an endpoint crew scheduled. Not "gets one and we will sort it out." Does not get one. Every exception I ever granted to that rule cost me more than holding it would have.

Acceptance Tests Written Once, Parameterized Forever

Because sites are environments, the acceptance suite is written against the manifest rather than against a site.

@pytest.mark.parametrize("ep", endpoints_of_class("surveillance"))
def test_stream_survives_uplink_failure(ep, site):
    with degrade_uplink(site, "SW-A"):
        assert stream_continuous(ep.address, seconds=120)
        assert resolved_tag(ep.address) == ep.tag

That second assertion is the one I care about most. It checks that the device answering at an address is the device the design says should be there. Physical-world deployments generate transposition errors constantly; two ports swapped during termination, and a suite that only tests function will pass happily on a swapped pair. You will find out during an incident, when someone pulls up the wrong view.

Phase One Writes the Template Whether You Intend It To or Not

The procedures we developed during the initial rollout became the template for every later phase of the same multi-year program. That was mostly not deliberate. It happened because those procedures were the only written record of why a given check existed, and later teams adopted them rather than rediscover the reasoning.

Which is worth saying plainly: your first phase is authoring the standard for everything that follows, and the artifacts that survive are the executable ones. Narrative test procedures rot. Nobody reads the PDF. A parameterized suite and a manifest schema get run, and when someone changes them, the change is visible.

The rework we avoided in later phases did not come from testing harder. It came from the fact that the definition of "done" for a site was identical in phase three and phase one, and a new engineer could read it in an afternoon.

Where It Still Breaks

I do not want to oversell this. Two things reliably escape the model.

The first is anything genuinely site-specific: a structure with an unusual pathway, an interface to an older system that predates the standard. Those exist. The discipline is to name them as exceptions with their own schedule, not to loosen the standard so they fit inside it. One exception absorbed into the template contaminates every site that follows.

The second is that batching increases the blast radius of a design defect. When configuration is generated, a bad rule ships everywhere at once. That is the trade you accept, and the mitigation is the same one teams use: a canary. The first site through each stage gets scrutiny nobody else gets, and nothing promotes behind it until it clears.

Coverage was never my constraint. Order was.