{"id":158,"date":"2026-08-07T09:00:00","date_gmt":"2026-08-07T09:00:00","guid":{"rendered":"https:\/\/vectormatch.dev\/blog\/?p=158"},"modified":"2026-08-07T11:05:24","modified_gmt":"2026-08-07T11:05:24","slug":"rate-limiter-only-broke-in-production","status":"publish","type":"post","link":"https:\/\/vectormatch.dev\/blog\/rate-limiter-only-broke-in-production\/","title":{"rendered":"The Rate Limiter That Only Broke in Production"},"content":{"rendered":"

Six parallel workers, six independent counters, one provider cap \u2014 and a test suite that never caught it because every test ran sequentially.<\/p>\n

\n

<\/svg>The short answer<\/p>\n

I deployed a Bottleneck.js rate limiter capped at 2 req\/s per ATS platform. It passed every test. In production, six parallel Inngest workers each got their own in-process limiter, so the aggregate hit 12 req\/s \u2014 six times the cap. The provider returned 429s within minutes. The fix: move the limiter state into Redis so all workers share a single global counter. The bug was not in the code but in my mental model of what ‘rate limited’ means across processes.<\/p>\n<\/div>\n

The Bug: 2 req\/s in Tests, 12 req\/s in Production<\/h2>\n

The rate limiter was simple and correct \u2014 on paper. A Bottleneck.js<\/a> instance per ATS platform (Greenhouse, Lever, Ashby, SmartRecruiters, Workable, Recruitee), each configured with minTime: 500<\/code> milliseconds and maxConcurrent: 1<\/code>. That enforces a maximum of 2 requests per second per platform, which is the documented rate limit for most ATS public APIs. The code was clean, the tests were green, and the first production deploy blew through the cap within 90 seconds.<\/p>\n

The problem was not in the limiter configuration. It was in where the limiter lived. Each Inngest worker process \u2014 and I run six in parallel \u2014 called getLimiter("greenhouse")<\/code> and got back its own Bottleneck instance with its own internal counter. Worker A tracked its own 500ms spacing. Worker B tracked its own. So did C, D, E, and F. Individually, each was perfectly rate-limited at 2 req\/s. Collectively, they were hitting Greenhouse at 12 req\/s, which is well above the threshold where ATS platforms start returning HTTP 429 or, worse, silently IP-banning the source.<\/p>\n

Why Every Test Passed (and Why That Was the Real Bug)<\/h2>\n

The test suite had full coverage of the rate limiter. It verified that consecutive calls were spaced at least 500ms apart. It verified that different ATS sources got different limiter instances. It verified cleanup and reconnection. Every test passed. Every test was also completely useless for catching this bug, because every test ran inside a single Node.js process.<\/p>\n

The Sequential Test Trap<\/h3>\n

Here is the test that gave me false confidence:<\/p>\n

it(\"enforces the minTime rate limit between calls\", async () => {\n  const limiter = getLimiter(\"test-rate\");\n  const timestamps: number[] = [];\n  const recordTimestamp = () => {\n    timestamps.push(Date.now());\n    return Promise.resolve();\n  };\n\n  \/\/ Schedule 3 calls \u2014 they should be spaced at least ~500ms apart\n  await limiter.schedule(recordTimestamp);\n  await limiter.schedule(recordTimestamp);\n  await limiter.schedule(recordTimestamp);\n\n  expect(timestamps).toHaveLength(3);\n  const gap1 = timestamps[1] - timestamps[0];\n  const gap2 = timestamps[2] - timestamps[1];\n  expect(gap1).toBeGreaterThanOrEqual(450); \/\/ ~500ms - 50ms tolerance\n  expect(gap2).toBeGreaterThanOrEqual(450);\n});<\/code><\/pre>\n
This test is correct for what it tests: a single limiter instance in a single process enforces 500ms spacing between sequential calls. But the production bug was not about sequential calls to one limiter. It was about six separate limiter instances in six separate processes all hitting the same external API concurrently. No unit test can reproduce that, because unit tests do not spawn six Node.js processes and point them at the same rate-limited endpoint.<\/p>\n
The Mock That Hid the Problem<\/h3>\n
The integration tests for the ATS adapters mocked the rate limiter entirely:<\/p>\n
vi.mock(\"@\/lib\/jobs\/poller\/rate-limiter\", () => ({\n  getLimiter: () => ({\n    schedule: (fn: () => Promise<unknown>) => fn(),\n  }),\n}));<\/code><\/pre>\n
This mock executes the scheduled function immediately with no delay \u2014 which is correct for testing the adapter logic (Zod validation, normalisation, error handling) but means the integration tests never exercised the rate limiter at all. The rate limiter was tested in isolation, the adapters were tested with the rate limiter mocked out, and I never tested the two together across multiple processes. That gap is where the bug lived.<\/p>\n
\"Sequential
Sequential tests verify one limiter in one process. Production runs six limiters in six processes. The gap is where the bug lived.<\/figcaption><\/figure>\n
The Wrong Mental Model: Process-Local Is Not Global<\/h2>\n
The root cause was not a coding error. It was a mental model error. My implicit assumption was that "rate limited" means "the API will not receive more than 2 req\/s." What the code actually guaranteed was "no single process will send more than 2 req\/s." Those are only equivalent when there is exactly one process.<\/p>\n
The Assumption Stack<\/h3>\n
Here is the chain of assumptions that made the bug invisible:<\/p>\n