Turning 40,000 support calls a month into a product backlog

The situation

Quality analysts listened to about 2% of calls, scored them against a rubric, and wrote a monthly summary. Product and operations both suspected the summary reflected which calls got sampled more than what customers were experiencing, but nobody could argue with it because there was no alternative.

The constraint

Three languages, frequent code-switching mid-sentence, background noise, and audio quality that varied by channel. Recordings contain personal and payment information. And a prior vendor pilot had produced confident-sounding summaries that the quality team spot-checked, disagreed with, and abandoned — so the bar for trust was high and the memory was recent.

What we built

The graded set, first

Before any pipeline, we sat with two quality leads and built 180 calls with agreed expected outputs: the reason for contact, whether it was resolved, the products mentioned, the compliance flags, and the customer sentiment at open and close. That set became the acceptance criteria, the regression suite, and the thing that ended arguments about whether a change helped.

Transcription tuned for the actual audio

Speaker diarisation to separate agent from customer, language identification per segment, and a custom vocabulary of product names, plan names and local place names. Redaction of card numbers and identifiers happens before text leaves the processing boundary, and the redaction is checked against its own test set.

Extraction with structure, not summarisation

The model is asked for a defined schema with explicit "not stated" options, and it must cite the transcript span supporting each field. Output is validated against the schema; a failure is retried once and then queued for review. Free-text summarisation exists, but no downstream metric depends on it.

Evaluation in the pipeline

Every prompt, model or vocabulary change runs the graded set in CI and reports per-field F1 plus a citation-faithfulness check. Two model upgrades that looked better in demos were measurably worse on two fields, and we caught both before release.

A console where disagreement is data

Quality leads review flagged and sampled calls side by side with the transcript, correct fields inline, and every correction is versioned. Corrections feed the graded set, so the evaluation baseline gets stronger the more the tool is used.

What changed

Every call is now analysed rather than a small fraction sampled, and the monthly themes report is a query rather than a project. The output that mattered most was unglamorous: a ranked list of contact reasons that turned out to be dominated by two self-service gaps, both fixed within a quarter.

What we would do differently

We under-invested in the redaction test set early, treating it as a compliance checkbox rather than a model with its own failure modes. It needed the same graded-set discipline as the extraction, and we retrofitted that in month four.

Outcomes

Calls reviewed

a small sample → 100% analysed

Extraction accuracy on the graded set

a substantial improvement

Time to a themed monthly report

weeks → same day

Client identity withheld under a mutual NDA. Figures are illustrative — rounded and directional, meant to show the shape of the change rather than an audited result. We will walk through the real numbers, and how they were measured, under NDA on a call.