Shift-Left Without Losing the Audit Trail: Test Automation for Regulated Surgical Software

In most software, a red test is a bug. On the systems I work on, a red test can be a patient-safety signal. That one difference reshapes almost every decision you make when you sit down to design a quality strategy.

I've spent more than a decade in software quality, most of it around medical device software: robotic-assisted surgery, surgical simulation, and clinical education platforms. The engineering is interesting on its own. What makes it genuinely hard is that every test, every pipeline, and every release has to satisfy two audiences at the same time. Engineers want fast feedback. Regulators want traceable evidence that the software does exactly what its requirements say, and nothing dangerous besides.

For a long time, teams treat those as opposing forces. You either move fast or you stay compliant; pick one. I don't think that trade-off is real anymore, and most of what I do now is prove that in practice.

Here is the pattern I've watched sink more than one otherwise capable team.

You automate your tests. Good. Your pipeline goes green, everyone feels productive. Then, separately, someone opens a document and starts writing the validation record: which requirement each test covers, what the acceptance criteria were, what the result was, who reviewed it. That document is what an auditor actually reads. And it lives in a different system from the tests it describes.

So the two drift. A test gets renamed, and the doc still references the old name. A requirement changes and the automation updates, but the traceability matrix doesn't, or the other way around. Nobody notices until an audit or a release review, at which point you are reconstructing history under time pressure, which is the worst possible condition for accuracy.

The cost here isn't just wasted hours. It's that the audit trail stops being trustworthy, and in regulated software an untrustworthy audit trail is close to worthless. You end up paying twice: once to run the tests, once to prove you ran them, and the second payment keeps bouncing.

The fix that has worked best for my teams is boring in the best possible way. Stop treating traceability as documentation, and start treating it as test metadata.

Under standards such as IEC 62304, you need to show for each software requirement and each identified hazard that a verification exists and that it passed. There is no rule that says a human has to type that mapping into a spreadsheet by hand. So we attach the mapping to the test itself: 

import pytest

@pytest.mark.requirement("SRS-1420")          # links to a software requirement
@pytest.mark.risk("HAZ-07", level="high")     # links to a hazard in the risk file
def test_instrument_motion_halts_on_fault(surgical_sim):
    surgical_sim.inject_fault("encoder_dropout")
    surgical_sim.command_motion(axis="wrist", degrees=15)

    # Safety requirement: motion must stop within the specified window
    assert surgical_sim.motion_state == "halted"
    assert surgical_sim.time_to_halt_ms <= 100

Nothing exotic is happening here. But this test no longer just checks behavior. It knows which requirement it verifies and which hazard it mitigates, and that knowledge travels with the code through every refactor, rename, and merge. When the test moves, its traceability moves with it, because they are the same artifact.

A small conftest.py hook collects those markers at collection time and emits them alongside the results. The requirement-to-test mapping stops being something a person maintains and becomes something the test suite reports about itself.

Once the metadata lives on the tests, the CI pipeline can generate the traceability record instead of a human writing it after the fact. That changes the economics entirely.

verify:
  stage: test
  script:
    - pytest --junitxml=results.xml -m "requirement"
    - python tools/build_trace_matrix.py results.xml requirements.csv > trace_matrix.html
  artifacts:
    paths:
      - trace_matrix.html
    when: always

risk-coverage-gate:
  stage: verify-coverage
  script:
    # fail the build if any high-risk requirement has no passing test behind it
    - python tools/check_risk_coverage.py results.xml risks.csv --min-level high

The first job builds the traceability matrix as a pipeline artifact, versioned against the exact commit that produced it. It is dated, reproducible, and it never disagrees with the code, because it was generated from the code.

The second job is the one I actually lose sleep over, in a good way. It fails the build when a high-risk requirement has no passing test behind it. That single gate converts a coverage gap from something you discover during an audit into something you discover at ten in the morning on a Tuesday, while the person who introduced it is still at their desk and remembers why. Cheap to fix now, expensive to fix later. Moving that discovery earlier is most of the value.

A trap on the other side of this is treating every requirement as equally sacred. That sounds responsible, and it is actually a way to run out of time.

Risk management thinking, in the ISO 14971 sense, gives you a defensible way to spend your effort unevenly. A label that renders in the wrong font and an instrument that fails to halt on a fault are both technically "defects." They are not remotely the same defect, and no honest test strategy pretends they are. The high-severity paths get exhaustive automated coverage, boundary analysis, fault injection, and repeated runs under load. The cosmetic paths get a reasonable check and move on.

This is also how I decide what to automate first when a team is drowning. Sort by risk, not by whatever is easiest to script. The most valuable test to automate is usually the one guarding the hazard you would least want to explain in an incident review.

"Shift-left" gets used as if it just means "test a bit earlier." In a regulated setting, it means something more specific and more demanding: get the requirement, the risk assessment, and the acceptance criteria into the same conversation before the code exists.

When a QE engineer is in the room while a requirement is still being written, they ask the questions that are painful to answer later. How do we observe this behavior from outside the system? What is the measurable threshold for "safe"? What happens on the fault path, not just the happy path? Those questions shape the design so it is testable and traceable by construction, instead of retrofitting testability onto something that was never built to expose its own state. Retrofitting works. It just costs several times more and produces worse tests.

I'll be honest that the technical part is the easy part. The hard part is the humans.

Engineers who come from unregulated web or consumer backgrounds often experience the documentation and traceability as bureaucracy, a tax that slows down real work. I don't blame them, because when traceability is maintained by hand, it genuinely is that. My job leading globally distributed teams is less about writing frameworks and more about changing that felt experience.

The turn happens the first time someone watches a risk-coverage gate catch a real gap that would otherwise have shipped. Suddenly the process isn't paperwork; it's a teammate that caught something before a patient could. That is a very different feeling, and it is the moment adoption stops being something I have to push. Across time zones, that shift has to happen locally, over and over, which is why I care more about a few visible saves than about any policy memo. People adopt what they have seen work, not what they have been told to do.

None of this makes regulated software fast. It is slow for reasons that are mostly good ones, and I would be suspicious of anyone selling a shortcut around design controls in a system that moves surgical instruments inside a human body.

What it does remove is the self-inflicted part. For years I accepted the double-bookkeeping, the drift, and the audit-time scramble as simply the cost of being regulated. Most of it wasn't. It was tooling I hadn't built yet. Once the tests carry their own traceability and the pipeline emits the evidence, the compliance record stops being a separate deliverable and becomes a byproduct of doing the engineering well. You still move deliberately. You just stop paying for the same work twice.

That, to me, is the whole game in this domain: make the safe path and the fast path the same path, so nobody has to choose between them under pressure.