A reproducible pipeline for a diagnostics lab that had outgrown its scripts

The situation

The lab's analysis ran on a set of shell and R scripts that had grown with the science. They worked, and the people who wrote them knew which flags to use. But results were assembled by hand at three points, reference data versions were whatever had been downloaded most recently, and re-running an analysis from six months earlier meant reconstructing an environment from memory.

The trigger was an accreditation review that asked a reasonable question: show us how this specific reported value was produced.

The constraint

We could not stop the lab. Sample volume was growing, clinical turnaround commitments were contractual, and the scientific team — correctly — would not accept a platform that made them slower or took the analysis logic out of their hands.

What we built

Instrument capture that does not depend on a person

Watchers on the instrument output shares register every run, compute checksums, record the instrument, operator, kit lot and reagent batch, and move raw files to immutable object storage. Nothing downstream reads from a share again.

Pipelines as versioned artefacts

The existing analysis logic was wrapped, not rewritten, into containerised Nextflow workflows. Each pipeline version pins its container digests and its reference-data release. A run record stores the pipeline version, parameters, inputs and outputs, so any result can be regenerated from its identifier alone. The scientists still own the workflow code; it now lives in a repository with review and tags.

Provenance as a first-class object

Every derived value carries a lineage chain back to the raw file, the pipeline version and the reference release. The accreditation question that started the project is now a query.

Quality gates in the flow, not at the end

Per-assay acceptance criteria — coverage, contamination estimates, control performance — run as automated checks. A failing sample is flagged and routed for review before it consumes analyst time. Trends across control samples are charted, so drift shows up as a slope rather than a surprise.

Reporting into the systems clinicians already use

Structured results flow to the LIS and to ordering systems over HL7 v2 and FHIR, with the narrative report generated from the same structured data that feeds the interface. Sign-out remains a human step, with an explicit electronic signature and an unambiguous amendment path.

What changed

Turnaround more than halved, most of it recovered from queueing and manual handling rather than compute. More importantly for the lab, a result is now a reproducible object with a history, which is what the accreditation body, and any future clinical dispute, actually needs.

What we would do differently

We introduced strict schema validation on the sample manifest earlier than the lab's upstream partners could support, and spent several weeks handling rejected files by hand. A quarantine-and-repair path should have shipped in the same release as the validation, not two sprints later.

Outcomes

Median turnaround, sample to report

more than halved

Pipeline runs reproducible from ID alone

100%, by design

Manual transcription steps removed

nearly all

Client identity withheld under a mutual NDA. Figures are illustrative — rounded and directional, meant to show the shape of the change rather than an audited result. We will walk through the real numbers, and how they were measured, under NDA on a call.