/en/research · what we measured
In our candidate matching, nearly half of what reaches the expensive LLM stage is garbage. We tried GPU rerankers, CrossEncoder fine-tuning and feature combos to filter it cheaper. The simplest signal won — skill overlap. Here are the numbers, what worked, and why text lost to skills.

Anyone who has built candidate-to-vacancy matching on top of "vector search + an LLM judge" eventually has an awkward conversation with the invoice. The LLM stage is the most expensive part of the pipeline, and it turns out to spend a good half of its time on candidates a human would reject in a second. We decided to measure exactly how bad it was and to cut the garbage before the expensive stage — ideally without adding yet another heavy model. The story is instructive: the cheapest signal won, while the fancy GPU rerankers and fine-tuning lost.
Our pipeline is classic: cosine vector search collects candidates, a reranker cuts some, the rest goes to the LLM, which returns a matched verdict from 0 to 1 and explains it. We dumped the state on a testing copy of the database and looked at the real numbers.
matched < 0.70 — i.e., they are garbage. Nearly 36,000 wasted LLM runs and just as many junk records in the database.The obvious question: why doesn't the reranker cut this? Because the existing cheap scores barely tell garbage from good. We computed the AUC for each — the ability to separate a "good" candidate (matched ≥ 0.70) from "garbage":
matchedBatch) — AUC only 0.60.That is why half of the garbage cruises straight to the LLM. We needed a signal that actually separates.
We framed the task like engineers, not "predict a score" but a cheap filter before the LLM. Two metrics:
The data for this is rich and balanced: 76,188 candidates with an LLM verdict, roughly half good and half garbage. No fine-tuning is needed to start — as our CTO put it, "first take off-the-shelf models and measure which one separates best."
We didn't guess — we ran a bake-off on the same sample:
BAAI/bge-reranker-v2-m3 (multilingual, ~568M) and cross-encoder/mmarco-mMiniLMv2-L12.The headline table. AUC — separate a good candidate from garbage (higher is better):
| Scorer | Runs on | AUC |
|---|---|---|
| cosine (current 1st stage) | CPU | 0.56 |
matchedBatch (current reranker) |
CPU | 0.60 |
| mmarco (text) | GPU | 0.56 |
| bge-reranker-v2-m3 (text) | GPU | 0.60 |
| MaxSim over skills | CPU | 0.75 |
The result is counterintuitive: heavy text rerankers on GPU (0.56–0.60) lost to simple skill overlap (0.75), which runs on CPU for pennies.
What else didn't work:
matched < 0.70. But we trained on the "verified" subset, which is almost all positives, and mined fake negatives instead of using the real ones. The model learned from a contradictory signal from mined "negatives," some of which are actually fine matches. That's not a verdict on the method — it's a botched setup on our side. By then off-the-shelf rerankers had already lost to skills on the full labeled set, so we didn't push a properly-trained CrossEncoder on the real negatives — an open question. The lesson landed on us: a broken training run is not a valid negative result — especially when the data was there and we picked the wrong slice.A warning for anyone measuring something similar. On our first pass we computed the correlation on "verified" offers only — and cosine looked completely useless there. That's a range-restriction artifact: 82% of those offers had been selected by cosine itself, so they all already have high cosine. Within a cosine-selected set, cosine has no power left to re-rank. Measuring a retriever's quality on the very data it selected is a classic way to fool yourself. Only the full candidate set, including the ones that were cut, gave the right picture.
The reason is simple once you think about it. A text reranker is trained to answer "is this passage relevant to the query." But all our candidates are already on topic — vector search selected them, they all look "relevant" to the reranker, and it can't tell them apart. The LLM verdict is about something else: does the candidate cover the vacancy's skills and requirements. That is exactly what skill overlap captures directly.
We confirmed this with one more measurement: the overall LLM verdict is almost entirely determined by the fraction of covered mandatory requirements — the AUC between "mandatory-requirement coverage" and "good" was 0.998. So the target is to predict requirement coverage, and the skill signal does it most cheaply. We even tested the CTO's hypothesis — take skills only from mandatory requirements (drop the optional ones from the filter). Sound in principle, but the numbers gave nothing: 0.682 vs 0.679 — a good candidate covers both mandatory and optional, so the restriction changes nothing.
We shipped a MaxSim gate before the database write. The logic at the selection stage: compute the skill overlap between the vacancy and the candidate; if it is below a threshold, the candidate is simply not created as an offer and never goes to the LLM.
The threshold is a single environment variable (0 disables it):
What it gives in practice:
A bonus that surfaced along the way: since the LLM verdict is requirement coverage, for each mandatory requirement you can see which skill is missing. That turns into a product feature — highlight the gap right in the offer and tell the recruiter what to ask about during screening.
Sometimes the best improvement to matching isn't a smarter model — it's a more honest question to your data.