Evals before merge: the golden set behind an AI matching pipeline

TL;DR: Before I let myself touch the AI matching pipeline of a job-matching product I run, I built an evaluation harness and a golden set: 193 human-judged candidate and job pairs, almost all of them taken from approve and reject decisions I had already made in the admin panel. Every change to the pipeline (model, prompt, threshold, weight, layer) now has to run the same command and report the same metrics before merge: P@10, strict P@10, NDCG@10, MRR, Recall@200, and violations@10, which must be zero. Three refactor and retrieval variants in a row landed within noise of the baseline, and a calibration experiment I was sure would work proved that it could not. The harness is a small CLI I wrote myself, because the metrics that matter are tied to how the product shows results, and the labels already lived in my own database.

The problem: a pipeline I could change but not measure

The matching pipeline takes a candidate profile and finds jobs worth showing, and the reverse. It has the usual layers of a modern retrieval system: vector retrieval with embeddings, hard filters for things like excluded technologies and seniority, an LLM pass that scores each surviving pair, and a composite score that orders the final list. Every one of those layers has knobs. I had a plan for a two-stage redesign with a pre-filter, a wider retrieval window, a reranker and a structured scoring prompt, and each step sounded like an obvious improvement.

That was the trap. An LLM pipeline is non-deterministic in the small and opinionated in the large. Change the prompt and some pairs move up, some move down, and the three you happen to eyeball look better. I had no way to say whether the list as a whole got better, and a job board lives or dies on the first ten results a person sees. So the redesign plan got a milestone zero that shipped nothing to users: an eval harness, a baseline, and a rule. No pipeline change merges without an eval result attached.

In the industry this is called LLM evals: a repeatable measurement of an LLM-backed system against a fixed set of judged examples, run before a change ships, the way a test suite runs before a deploy. The difference from unit tests is that the output is a score on a scale, so you also need to know how big a difference has to be before it means anything.

What is a golden set, and where do the labels come from?

A golden set is the fixed collection of inputs with trusted, human-assigned answers that every variant is scored against. For a ranking system, one row is a pair (this candidate, this job) and a grade. I used a three-point scale:

• 2 - I would show this without hesitation.
• 1 - borderline, defensible, not embarrassing.
• 0 - must not be shown, with a required reason: excluded technology, language, location, missing skill, seniority, salary, preferences, or other.

The expensive part of a golden set is usually labelling. I was lucky, or rather an earlier decision paid off: the admin panel had been recording every human approve and reject decision with a timestamp and a source, precisely so it could be measured one day. The first export pulled 189 human decisions plus a handful of weak positives from real clicks, for 193 pairs across 26 candidate profiles, split roughly half and half between good matches and rejections. Automated approvals were excluded on purpose. A decision a cron job made is not a judgement.

For the gaps, the harness writes a labelling pool: the top thirty jobs per candidate from the current pipeline, minus pairs that already have a label. I fill in a grade in a spreadsheet and import it back. The plan budgeted an hour or two of labelling; because the admin panel had already covered most pairs, the first round was 25 pairs and about ten minutes.

Two details keep this safe to commit. The golden set stores only opaque IDs, grades and reasons, no personal data. The labelling worksheet, which shows readable titles and contact details for convenience, never enters version control.

// illustrative shape - one golden row
{ "candidate": "<uuid>", "job": "<uuid>",
  "label": 0, "reason": "location",
  "source": "admin_decision", "labeled_at": "<iso>" }

Which metrics matter for an LLM ranking pipeline?

I picked the metrics from what a user actually experiences, then added one that exists only to protect them.

The ship gate deserves a sentence of its own. Precision can trade off against other things; a job in the wrong country or requiring a technology the candidate explicitly excluded cannot. Three of the rejection reasons count as hard violations, and a single one inside a top ten fails the run regardless of how good the averages look. The same rules also live in a set of end-to-end fixtures (synthetic pairs like a candidate who excluded a framework against a job built on it), which grew from 22 to 30 over the redesign and guard the safety filters independently of the ranking metrics.

Coverage is the metric people forget. If a new variant pulls unjudged pairs into the top ten, the other metrics are computed only over the judged part and can look stable while hiding a change. Low coverage means label more before believing anything.

The options I considered for the harness

An eval framework or platform (promptfoo, LangSmith and similar). These are good at what they are built for: prompt-level test cases, an input, an output, and an assertion or an LLM-graded check, with dashboards and history. My unit of evaluation was different. It was a whole ranking per candidate, produced by retrieval, filters, an LLM pass and a weighted score together, judged with ranking metrics against labels that already sat in my database. Fitting that into a prompt-test shape meant exporting data, writing custom metrics anyway, and adding another processor for data derived from personal profiles, which comes with paperwork I avoid when I can. What remained for the tool to do was a dashboard, and a JSON file per run covered that.

LLM-as-a-judge, where a model grades the outputs instead of a person. It is cheap and scales, and it is a reasonable regression smoke test. I kept ground truth human, because the person approving matches knows the market and the candidates, and a judge model would share many blind spots with the scoring model it is judging.

Online A/B testing on users. With the traffic of a young product there is no statistical power; a test would run for months and still say nothing. I deferred it. What sits between offline evals and A/B is shadow mode: a new pipeline version writes its results tagged with a version, a human reviews both, and those decisions grow the golden set.

A small CLI of my own. Four subcommands: export labels from human decisions, write a labelling pool, import labels, and compute a baseline under a tag. The metrics are pure functions with no I/O, so they are easy to test. The harness is read-only: it reads the current ranking and the golden set and never writes to the results, so pointing it at real data carries no risk. Each run writes a timestamped, tagged JSON snapshot that is committed, so the history of every variant sits in the repo next to the code that produced it. This is what I chose.

// illustrative shape - one committed result snapshot
{ "tag": "m2", "golden_rows": 193, "profiles": 26,
  "macro": { "P@10": 0.535, "strictP@10": 0.497,
             "NDCG@10": 0.836, "MRR": 0.682, "coverage@10": 0.59 },
  "violations_at_10_total": 0 }

How do you evaluate an LLM ranking change before merge?

You run the same golden set through the baseline and the variant and read the difference against a noise threshold you set in advance. With a few hundred labels and a couple of dozen profiles, confidence intervals are wide, so I treat anything under five percentage points as noise. The target for a real improvement was ten points, directionally. Here is what the first redesign milestones produced:

None of the variants beat the baseline. For the first one, that was the goal: it merged three copies of scoring logic into one, fixed a skill normalizer that had drifted between two maps, and made re-matching preserve human decisions. The gate for a refactor is "not worse", and it passed. The interesting signal was elsewhere. Coverage fell from 1.00 to 0.66 because the fixed normalizer stopped rejecting pairs for falsely missing skills, the candidate pool grew from 225 to 360 pairs, and about a third of the new top ten had never been judged. The ranking metrics were stable over the judged part and silent about the rest, and the harness told me so instead of letting me read stability as success.

The second variant widened the pool further, to 519 pairs (+44%), and the third variant stayed within noise of it. The reranker in that milestone was still switched off behind a flag, so its effect was not in these numbers at all, which is exactly the kind of thing a results file with a tag makes obvious later. Without the harness I would have shipped all three with a story about why each one was better.

One more check rides along with every run: the distribution of raw embedding similarity across all stored pairs. The 10th to 90th percentile spread was 0.40 to 0.65, which confirmed an assumption in the plan about how much ranking signal similarity alone can carry. I wrote about why those raw scores mean so little on their own in tuning pgvector similarity thresholds.

The experiment that failed, and what it taught me

The redesign had an ambitious goal: make matching good enough to auto-approve the confident pairs and send only the uncertain ones to human review. The plan was to fit a logistic regression on human decisions, using the pipeline's signals as features, and sweep a threshold precision-first until auto-approval reached roughly 95 to 98 percent precision.

It did not get close. On 149 human-decided pairs with an LLM score, precision plateaued around 72 to 73 percent at every threshold. The weights were the real finding: keyword overlap carried most of the signal, and the LLM's overall score barely separated approved from rejected pairs. The part of the pipeline I was paying per call for was the part that told the human reviewer the least. The gate stayed unarmed, and the next steps became obvious: more labels, a reranker signal, and a benchmark of the scoring model on this same golden set before paying for a bigger one.

There are known limits to this whole setup, and I keep them written next to the numbers. The labels are pooled from what the pipeline already retrieved, so Recall@200 comes out at 1.0 by construction and says little until a retrieval change pulls in pairs from outside the pool. The baseline is slightly circular, because the human decided on rankings produced by the same pipeline. And the set is small. None of that makes it useless as a reference point for variants; it makes it a directional instrument, and I read it that way.

What I would tell someone doing this tomorrow

• Record human decisions before you need them. An approve or reject button that stores who, when and why is a golden set that grows during normal work. It turned two hours of labelling into ten minutes.
• Pick metrics from what the screen shows. Users see ten results, so measure the top ten. Add order (NDCG) and first good result (MRR) only because they answer questions precision cannot.
• Make the dealbreaker metric a gate. Averages trade off; a job in the wrong country does not. Zero, or it does not ship.
• Set the noise threshold before the first run. Otherwise every small wobble becomes a story. On a small set, five points was mine.
• Watch coverage. A variant that brings unjudged results into the top ten can look stable while hiding the change you care about.
• Version the results with the code. A tagged snapshot per run is cheaper than a platform and answers the question "what did we know when we merged this" months later.
• Expect most variants to lose. Three sensible changes in a row landed within noise. That is the harness doing its job.

The same discipline applies one layer down: an evaluation is only as good as the structured data the pipeline extracts in the first place, which is what tool use over parsing is about.