The Rate Limiter That Only Broke in Production

The Bug: 2 req/s in Tests, 12 req/s in Production

The rate limiter was simple and correct — on paper. A Bottleneck.js instance per ATS platform (Greenhouse, Lever, Ashby, SmartRecruiters, Workable, Recruitee), each configured with minTime: 500 milliseconds and maxConcurrent: 1. That enforces a maximum of 2 requests per second per platform, which is the documented rate limit for most ATS public APIs. The code was clean, the tests were green, and the first production deploy blew through the cap within 90 seconds.

The problem was not in the limiter configuration. It was in where the limiter lived. Each Inngest worker process — and I run six in parallel — called getLimiter("greenhouse") and got back its own Bottleneck instance with its own internal counter. Worker A tracked its own 500ms spacing. Worker B tracked its own. So did C, D, E, and F. Individually, each was perfectly rate-limited at 2 req/s. Collectively, they were hitting Greenhouse at 12 req/s, which is well above the threshold where ATS platforms start returning HTTP 429 or, worse, silently IP-banning the source.

Why Every Test Passed (and Why That Was the Real Bug)

The test suite had full coverage of the rate limiter. It verified that consecutive calls were spaced at least 500ms apart. It verified that different ATS sources got different limiter instances. It verified cleanup and reconnection. Every test passed. Every test was also completely useless for catching this bug, because every test ran inside a single Node.js process.

The Sequential Test Trap

Here is the test that gave me false confidence:

it("enforces the minTime rate limit between calls", async () => {
  const limiter = getLimiter("test-rate");
  const timestamps: number[] = [];
  const recordTimestamp = () => {
    timestamps.push(Date.now());
    return Promise.resolve();
  };

  // Schedule 3 calls — they should be spaced at least ~500ms apart
  await limiter.schedule(recordTimestamp);
  await limiter.schedule(recordTimestamp);
  await limiter.schedule(recordTimestamp);

  expect(timestamps).toHaveLength(3);
  const gap1 = timestamps[1] - timestamps[0];
  const gap2 = timestamps[2] - timestamps[1];
  expect(gap1).toBeGreaterThanOrEqual(450); // ~500ms - 50ms tolerance
  expect(gap2).toBeGreaterThanOrEqual(450);
});

This test is correct for what it tests: a single limiter instance in a single process enforces 500ms spacing between sequential calls. But the production bug was not about sequential calls to one limiter. It was about six separate limiter instances in six separate processes all hitting the same external API concurrently. No unit test can reproduce that, because unit tests do not spawn six Node.js processes and point them at the same rate-limited endpoint.

The Mock That Hid the Problem

The integration tests for the ATS adapters mocked the rate limiter entirely:

vi.mock("@/lib/jobs/poller/rate-limiter", () => ({
  getLimiter: () => ({
    schedule: (fn: () => Promise<unknown>) => fn(),
  }),
}));

This mock executes the scheduled function immediately with no delay — which is correct for testing the adapter logic (Zod validation, normalisation, error handling) but means the integration tests never exercised the rate limiter at all. The rate limiter was tested in isolation, the adapters were tested with the rate limiter mocked out, and I never tested the two together across multiple processes. That gap is where the bug lived.

The Wrong Mental Model: Process-Local Is Not Global

The root cause was not a coding error. It was a mental model error. My implicit assumption was that "rate limited" means "the API will not receive more than 2 req/s." What the code actually guaranteed was "no single process will send more than 2 req/s." Those are only equivalent when there is exactly one process.

The Assumption Stack

Here is the chain of assumptions that made the bug invisible:

• Assumption 1: Bottleneck’s minTime: 500 enforces 2 req/s. True within a single instance.
• Assumption 2: getLimiter("greenhouse") returns the same limiter for all callers. True within a single process. False across processes.
• Assumption 3: Inngest worker functions share memory. False. Each worker is a separate Node.js process with its own module cache.
• Assumption 4: The test environment matches production topology. False. Tests run in one process; production runs six.

Each assumption is reasonable in isolation. The bug emerges from the composition: assumption 2 is true in tests (one process), which reinforces assumption 4, which hides the fact that assumption 3 is false in production. This is the classic shape of a distributed systems bug — the code is correct for the model I held, and the model was wrong.

The rate limiter code was correct. The tests were correct. My mental model of what "rate limited across six processes" means was wrong. The bug was in the gap between the model and the reality of distributed execution.

Why Inngest Made This Hard to See

Inngest abstracts away the concurrency model. You write a function, register it, and Inngest handles fan-out, retries, and parallelism. In development, Inngest runs functions in a single process — so the rate limiter works perfectly. In production, Inngest’s worker pool scales horizontally, and each worker is a separate process. The platform does not advertise this prominently, and the local dev experience gives you a mental model of "one process, one limiter" that is silently false in production.

This is not a criticism of Inngest specifically. Any job queue that scales workers horizontally (BullMQ, SQS + Lambda, Celery with multiple workers) has the same property. The rate limiter bug is a distributed systems bug, not an Inngest bug. But Inngest’s excellent local DX makes the single-process mental model particularly easy to accidentally hold.

The Fix: Move the Counter to Redis

The fix was conceptually simple: move the rate limiter’s state out of each process’s memory and into a shared store that all workers can see. I used Redis, via Bottleneck’s built-in ioredis datastore. This required zero changes to the calling code — limiter.schedule(() => fetch(url)) works identically whether the limiter is in-process or Redis-backed. The only change was in how the Bottleneck instance is constructed.

Before: In-Process (Buggy)

const limiters: Record<string, Bottleneck> = {};

export function getLimiter(atsSource: string): Bottleneck {
  if (!limiters[atsSource]) {
    limiters[atsSource] = new Bottleneck({
      minTime: 500,       // 500ms between requests = 2 req/s max
      maxConcurrent: 1,   // no parallel requests
    });
  }
  return limiters[atsSource];
}

Each process gets its own limiters object. Six processes = six independent counters. The 2 req/s cap is per-process, not global.

After: Redis-Backed (Correct)

let bottleneckConnection: Bottleneck.IORedisConnection | null = null;

function initRedis(): void {
  const redisUrl = process.env.REDIS_URL;
  if (!redisUrl) return; // in-process mode for dev/CI

  bottleneckConnection = new Bottleneck.IORedisConnection({
    clientOptions: redisUrl,
    Redis: IORedis,
  });
}

export function getLimiter(atsSource: string): Bottleneck {
  if (!limiters[atsSource]) {
    initRedis();

    if (bottleneckConnection) {
      // Distributed mode: Redis enforces the cap GLOBALLY.
      limiters[atsSource] = new Bottleneck({
        id: `ats:${atsSource}`,
        datastore: "ioredis",
        connection: bottleneckConnection,
        minTime: 500,
        maxConcurrent: 1,
        clearBatcherBeforeStop: true,
      });
    } else {
      // In-process fallback (dev, CI)
      limiters[atsSource] = new Bottleneck({
        minTime: 500,
        maxConcurrent: 1,
      });
    }
  }
  return limiters[atsSource];
}

The key addition is datastore: "ioredis" and the shared bottleneckConnection. Bottleneck uses Redis to store its internal state (the queue, the last-execution timestamp, the concurrency counter). When worker A schedules a request, it increments the Redis counter. Worker B sees that increment and waits. The cap is now global — 2 req/s across all six workers combined, not 2 req/s per worker.

The id: "ats:${atsSource}" field namespaces each ATS platform’s state in Redis, so Greenhouse and Lever still have independent rate limits. They just share the Redis connection.

The Fail-Closed Design

One critical design decision: if Redis is configured but unreachable, the limiter does not fall back to in-process mode silently. It throws. The caller catches the error and marks the ATS source as degraded. This is intentional. The failure mode of "no rate limiting" is an IP ban from the ATS platform, which is catastrophic and hard to recover from. The failure mode of "stalled ingestion" is self-healing — when Redis comes back, the poller resumes. I always choose the failure mode that self-heals over the one that makes things worse.

// If Redis is configured but unreachable, Bottleneck.schedule() throws.
// The caller catches this and returns a "network" error, which the
// circuit breaker treats as a degraded source.
//
// This is the correct failure mode: it's safer to stall ingestion
// than to risk uncoordinated requests that could trigger an ATS IP ban.

Dev/CI Behaviour

When REDIS_URL is unset (local dev, CI), the limiter falls back to in-process mode. This is correct: dev and CI run a single process, so the in-process cap is equivalent to the global cap. The fallback is logged with a warning so it is visible in CI output:

if (process.env.REDIS_URL) {
  console.warn(
    `[rate-limiter] REDIS_URL is set but Redis connection failed — ` +
    `using in-process limiter for "${atsSource}". ` +
    `Rate cap (2 req/s) is NOT enforced across multiple workers.`
  );
}

The Generalisable Lesson: Shared Constraints Must Live Outside the Process

This bug is not specific to rate limiters, Inngest, or Bottleneck. It is a general pattern that applies to any shared-resource constraint in a distributed system. If the constraint is a rate limit, a connection pool, a semaphore, or a quota — and the workload is handled by multiple processes — the constraint state must live in a shared store, not in process memory.

The Pattern

• Rate limiting: Use Redis, a token bucket in a shared store, or an API gateway. In-process limiters are only safe for single-process deployments.
• Connection pools: If six processes each open 10 database connections, you have 60 connections, not 10. Use a connection pooler like PgBouncer or a managed proxy.
• Concurrency limits: A "max 3 concurrent jobs" constraint enforced in-process allows 18 concurrent jobs across six workers. Use a distributed lock or a shared counter.
• Quota tracking: "100 API calls per hour" tracked per-process means 600 calls per hour across six processes. Track in Redis or a database.

The Testing Lesson

Sequential tests cannot catch concurrency bugs. If your production topology involves multiple processes hitting a shared resource, you need either:

1. Integration tests that spawn multiple processes and point them at the same resource (complex but realistic).
2. Chaos-style tests that simulate the failure mode (e.g., fire 12 concurrent requests and assert the external API receives at most 2 per second).
3. Load tests in a staging environment that matches production topology.

The minimum viable fix for my case was adding a test that spawns two Node.js child processes, each calling getLimiter("test"), and asserts that the combined request rate does not exceed 2 req/s. That test would have caught the bug. It would also have required a running Redis instance in CI, which is a trade-off worth making for any system where the rate limit exists to prevent IP bans.

What I Changed After This Bug

Beyond the Redis fix itself, I made three structural changes to prevent this class of bug from recurring. None of them are novel — they are standard distributed systems hygiene — but they were not in place when the rate limiter shipped, and their absence is what made the bug possible.

1. Document the Execution Topology Next to the Code

The rate limiter file now starts with a comment block that explicitly states the production topology and why it matters:

// When REDIS_URL is set, limiters use Bottleneck's Redis datastore so the
// 2 req/s cap is enforced GLOBALLY across all worker processes. This is
// critical for multi-worker Inngest deployments — without it, each process
// gets its own in-process limiter and the aggregate rate can exceed 2 req/s,
// risking IP bans from ATS platforms.

This comment exists because the next developer who reads this file — or me, three months from now — might not remember that Inngest runs six workers. The code should not require external knowledge to understand its constraints.

2. Fail-Closed on Infrastructure Dependencies

If Redis is configured but unreachable, the system stalls rather than running without rate limits. This is the opposite of the common "fail open" pattern where a missing dependency causes the safety check to be skipped. For rate limiting, failing open means "no rate limit," which means IP bans. Failing closed means "no ingestion," which means a delayed pipeline. I always choose the failure mode that does not cause irreversible damage.

3. Environment Parity for Shared State

Staging now runs with Redis configured, matching production. The previous staging setup ran without Redis (using in-process mode), which meant staging could not reproduce any bug that depended on the Redis/no-Redis boundary. Environment parity is not just about matching Node versions — it is about matching the presence and behaviour of every shared-state dependency.

The rate limiter bug was not a coding error. It was a mental model error — my assumption that ‘rate limited’ means ‘globally rate limited’ when the code only enforced ‘per-process rate limited.’ Six parallel Inngest workers, each with their own Bottleneck instance, hit the ATS API at six times the intended rate. The tests passed because they ran sequentially in a single process, which is the one topology where the bug does not manifest.

The fix was moving the limiter state to Redis via Bottleneck’s ioredis datastore, making the 2 req/s cap global across all workers. The calling code did not change. The failure mode was designed to fail closed — if Redis is down, ingestion stalls rather than risking an IP ban. This is the correct trade-off for any constraint where the failure mode of ‘no limit’ is worse than the failure mode of ‘no throughput.’

The generalisable lesson: any shared-resource constraint — rate limit, connection pool, concurrency cap, quota — must live outside the process when the workload is distributed. And sequential tests cannot catch bugs that only manifest under parallel execution across multiple processes. If your production topology involves more than one process hitting a shared resource, your tests need to reflect that.