Most of the Volume Never Reaches a Model

The Cost Arithmetic That Shaped Everything

Before I wrote a single line of the matching pipeline, I did the arithmetic. text-embedding-3-small costs $0.02 per million tokens. A job description, truncated to 8000 characters, is roughly 2000 tokens. That is $0.00004 per embedding — effectively free. gpt-4o-mini costs $0.15 per million input tokens and $0.60 per million output tokens. A Gate 3 evaluation sends about 3500 tokens in (system prompt plus the job and persona context) and gets about 200 tokens back (a structured verdict). That is $0.000525 in input plus $0.00012 in output, or roughly $0.0006 per call.

That ratio — 16x —is the reason the pipeline exists. If I sent every ingested job straight to the LLM, the cost would scale linearly with ingestion volume. If I filter first with SQL and vector math, the cost scales with candidate volume, which is a small fraction of ingestion volume. The gates are not an optimisation. They are the difference between a pipeline that costs dollars per day and one that costs cents.

The trade-off is precision. Every gate I add before the LLM is a filter that can reject a good match. The question is whether the filters are precise enough to justify the savings. The answer turned out to be yes, but only after I calibrated them against real data — and the calibration history is the story of three thresholds that each went through multiple revisions before they stopped rejecting things they should not have.

Gate 1: Tag Overlap in Postgres

The first gate is the cheapest thing I could build that still carries signal: set intersection on text arrays. Every job gets a set of extracted tags stored in a text[] column. Every persona has a set of must-have tags in another text[] column. Postgres has a GIN index on both. The overlap check is a single operator: p.must_have_tags && job.extracted_tags. The && operator returns true if the arrays share any element, and the GIN index makes it fast — the query planner turns it into an index scan, not a full table scan.

The threshold is not just any overlap. It is a minimum count. I require at least 2 shared tags:

export const GATE1_MIN_OVERLAP = Number(process.env.GATE1_MIN_OVERLAP ?? 2);

I started with 1. That was wrong. After two months of production data, I pulled 50 user-marked mismatches and found that 16 of them — nearly a third — had an overlap score of exactly 1. A single shared tag like docker or ci-cd was enough to pass a Java backend job to a TypeScript frontend persona, and Gate 3 then approved it because the prompt rule says "missing tags are a soft signal." The soft signal was too soft. Raising the minimum to 2 filtered those out at the SQL level before the LLM ever saw them.

There is also a stack-disjoint check layered on top of the overlap. It prevents a JavaScript persona from matching a Ruby job on process-noise tags — aws, docker, ci-cd — that appear in every engineering job description regardless of stack. The check is a set of AND NOT clauses in the SQL router: if the persona’s must-have tags include the JS family but the job’s tags do not, the candidate is excluded even if the overlap count is high. This is not a sophisticated filter. It is a blunt instrument. But it catches a class of false positives that the overlap count alone cannot.

Gate 2: Vector Similarity in pgvector

The second gate is where the architecture gets interesting. Every job and every persona has a 1536-dimensional embedding from text-embedding-3-small, stored in a vector(1536) column. pgvector provides the <=> operator for cosine distance, and I built an HNSW index on both the job and persona embedding columns:

CREATE INDEX "persona_embedding_hnsw_idx" ON "persona"
  USING hnsw ("persona_embedding" vector_cosine_ops);

CREATE INDEX "job_embedding_hnsw_idx" ON "job"
  USING hnsw ("job_embedding" vector_cosine_ops);

I chose HNSW over IVFFlat because the persona table is small — a few hundred rows — and HNSW does not require a training step or a lists parameter to tune. IVFFlat partitions the vector space into clusters and requires you to choose the number of clusters at index creation time. For a table that grows slowly and needs recall at small scale, HNSW is the simpler choice. The trade-off is build time and memory: HNSW indexes are larger and slower to build. For a few hundred vectors, neither matters.

The threshold was the hardest part. My initial value was 0.35 cosine distance — a guess based on synthetic data where distances clustered between 0.18 and 0.21. It rejected 100% of real job-persona pairs. Real embeddings have much wider variance: the minimum distance I saw was 0.45, the average was 0.61, the maximum was 0.74. I had calibrated against data that did not resemble production. I raised the threshold to 0.55.

Then I tightened it to 0.48, because a yield analysis showed that 77% of candidates (124 out of 160) had cosine distance above 0.45, and the 0.50–0.55 bucket had only a 2.9% approval rate at Gate 3. The 0.31–0.45 bucket had a 33% approval rate. The threshold was cutting into the wrong side of the distribution. I loosened it back to 0.50, then to 0.55, when a threshold simulation showed that at 0.50 only 3 matches survived from 347 global, embedded jobs across 3 personas, and at 0.55 there were 20 — a 6.7x increase, all high-quality web-dev roles.

Then I hit the cosine cliff. A perfect match at distance 0.5036 was being rejected for being 0.0036 over the threshold. The threshold was a hard gate — a binary cutoff — and it was throwing away candidates that were semantically identical to ones it accepted. So I changed the architecture. On D18, I made Gate 2 a rank signal instead of a gate. In rank-only mode, the threshold widens to a hard ceiling of 0.75 — anything below that gets inserted into the candidate queue, ordered by semantic distance. The LLM sees them all, ranked. The threshold is no longer a cliff. It is a soft signal in the composite score.

The composite score blends both gates:

compositeScore = (1 - exp(-0.4 * min(overlapScore, 5))) * 0.6
                + (1 - cosineDistance) * 0.4

Tag overlap gets 60% weight, vector similarity gets 40%. The overlap term is non-linear — 1 - exp(-0.4 * x) — so the first two matching tags carry more weight than the fifth and sixth. This mirrors how I actually read job descriptions: the first shared skill tells me more than the fourth.

Gate 3: The LLM Arbiter

Everything that survives Gates 1 and 2 goes to gpt-4o-mini. The call is direct — through the Vercel AI SDK’s generateObject with a Zod schema, not through a router or a fallback array. I never built a provider fallback. The single provider has not been the bottleneck; the cost has. Adding a fallback would add complexity without addressing the constraint that actually matters.

The model returns a structured verdict:

const gate3VerdictSchema = z.object({
  approved: z.boolean(),
  matchConfidence: z.number().min(0).max(1),
  matchReasoning: z.string().min(1).max(500),
  blockers: z.array(z.string()),
  workAuthRiskFlag: z.boolean(),
});

The system prompt is long — about 1500 tokens — because Gate 3 is the precision filter that catches what keyword matching cannot. It evaluates tech stack alignment, seniority fit, geo-fencing, work authorization, blocklist tags, and domain relevance. It has a hard rule: if the job title names a primary technology that is not in the persona’s must-have tags, reject immediately. "Java Developer" does not match a React persona, no matter how many shared process tags the overlap count shows.

There are three prompt variants — balanced, strict, and thorough — and the pipeline picks one at random per evaluation. I never measured whether one variant produces a materially different approval rate than the others. The A/B test is running, but I have not pulled the numbers. I never measured this.

The limit is 8 candidates per job. GATE_ROUTER_LIMIT = 8. That is the LIMIT in the SQL router query — the maximum number of personas that get inserted into the match queue for any single job. Most jobs generate zero candidates. The ones that do generate candidates rarely hit the cap. I never measured the exact distribution, but the cap exists as a safety net, not a regular constraint.

What I Measured and What I Did Not

The numbers I trust come from two sources: the calibration report I wrote after two months of production data, and the threshold simulation I ran on D16. The calibration report gave me the approval-rate-by-bucket data: 33% in the 0.31–0.45 cosine distance bucket, 2.9% in the 0.50–0.55 bucket. The simulation gave me the match-count-by-threshold data: 3 matches at 0.50, 20 at 0.55.

What I never measured is the end-to-end funnel volume. The code tracks it — there is an admin query called getFunnelStats that counts total jobs ingested, jobs passing Gate 0, Gate 1+2 candidates, and Gate 3 approvals over a rolling window. I built the query. I did not build a dashboard that shows me the numbers daily. I know the architecture filters aggressively — if it did not, the cost would be unsustainable — but I cannot tell you what percentage of ingested jobs produce zero candidates. I never measured this.

I also never measured the total daily LLM spend. I can compute the ceiling: if every job generated 8 candidates, the cost would be 8 × $0.0006 per job. But most jobs generate zero. The actual spend is somewhere between those two numbers, and I have not instrumented the pipeline to tell me where. This is a gap I should close. The cost arithmetic that motivated the architecture is also the arithmetic I should be measuring, not just computing.

The precision-versus-recall trade-off is the one I think about most. The 2.9% approval rate in the 0.50–0.55 cosine bucket tells me that Gate 2, at that threshold, was letting through candidates the LLM almost always rejected. That is wasted Gate 3 spend. The 33% approval rate in the 0.31–0.45 bucket tells me that the tight threshold was also blocking candidates the LLM would have approved. That is lost recall. The D18 re-architecture — making Gate 2 a rank signal instead of a hard gate — was the resolution: let everything through below the hard ceiling, let the LLM decide, and use the cosine distance for ordering instead of exclusion. The cost goes up because more candidates reach Gate 3. The recall goes up because the cliff is gone. The precision stays the same because the LLM is still the final filter.

The Generalisable Lesson

The architecture is not specific to job matching. It is the standard pattern for any pipeline where a cheap, precise filter can reduce the volume that reaches an expensive, nuanced evaluator. The pattern has three properties:

The expensive stage should be last. Every filter before the LLM is a filter that costs fractions of a cent. The LLM call costs 16x more than the embedding and runs on every candidate that survives. Putting the LLM first — "just ask the model to evaluate everything" — is the obvious architecture and the wrong one. The cost scales with input volume, not with match volume.

Every threshold needs calibration against real data. My initial cosine distance threshold of 0.35 was calibrated against synthetic data and rejected 100% of real pairs. The 0.48 threshold was calibrated against two months of production data but over-filtered borderline matches. The 0.55 threshold was calibrated against a threshold simulation and struck the right balance. Each revision was a response to a measurement, not a guess. If you cannot measure the approval rate by bucket, you cannot tune the threshold. You are just moving a slider in the dark.

Hard thresholds create cliffs. Use rank signals when you can. The cosine cliff — rejecting a match at 0.5036 for being 0.0036 over the threshold — was the dumbest failure mode in the pipeline. The fix was not a better threshold. It was demoting the threshold from a gate to a ranking signal. The LLM sees more candidates, ordered by semantic distance, and makes the call itself. The cost increase is bounded by the candidate cap. The recall improvement is not.

The pipeline is not finished. The gaps I know about — no daily funnel dashboard, no per-variant approval rate, no actual spend measurement — are the next things to instrument. The architecture is sound. The observability is not. But the core decision, that most of the volume should never reach a model, has held up under every revision I have made to the thresholds above it.

The 3-gate pipeline exists because the LLM is 16x more expensive than the embedding and the cost scales with candidate volume, not ingestion volume. Gate 1 — a GIN-indexed tag intersection in Postgres — costs effectively nothing and filters out jobs with no stack overlap. Gate 2 — an HNSW vector similarity search in pgvector — costs the embedding generation and ranks the survivors by semantic distance. Gate 3 — a gpt-4o-mini call — costs $0.0006 per evaluation and sees only the candidates that survived both.

The thresholds took three revisions to get right. The initial cosine distance of 0.35 was calibrated against synthetic data and rejected everything. The 0.48 threshold was calibrated against real data but over-filtered borderline matches. The 0.55 threshold struck the balance but created a cliff — a match at 0.5036 rejected for being 0.0036 over. The D18 re-architecture demoted the threshold from a hard gate to a rank signal, letting the LLM decide on everything below a 0.75 hard ceiling. The cost went up. The recall went up with it.

The architecture is sound. The observability is not. I never built a daily funnel dashboard. I never measured the total LLM spend. I never checked whether the three prompt variants produce different approval rates. The next step is not another threshold revision — it is instrumentation. You cannot tune what you do not measure, and I have been tuning longer than I have been measuring.