Problem
High-demand restaurants release reservations at scheduled times each week, and the window to book one is measured in seconds. I wanted to build a system that could monitor availability across many concurrent users, detect openings the moment they appeared, and execute bookings faster than a person could click through a website. The challenge was not just speed. It was reliability at scale. When fifty users are monitoring the same handful of venues simultaneously, the system needs distributed coordination to prevent double-bookings, has to tolerate upstream API changes gracefully, and must avoid correlated traffic patterns that would get the entire user base flagged. Every subsystem had to assume multiple worker instances running in parallel, because horizontal scaling of a single BullMQ worker was the only path to acceptable latency during weekly release windows.
Approach & Architecture
I architected SeatWatch as a Turborepo monorepo with four independently deployed services on Railway (a React 19 SPA, an Express REST API with sixteen user-facing route modules, a BullMQ worker with five independently tuned queues, and a separate Next.js 16 SEO content site) plus a shared packages/shared that owns types, Zod schemas, the browser identity pool, and the generated Prisma client.
The booking path is a dual-strategy engine. The primary path makes direct HTTP calls to the reservation API and completes in sub-second time; on recoverable errors (403, 404, or unrecognized response shapes) the worker transparently falls back to Patchright, a headless-Chrome driver that takes 2–5 seconds but tolerates UI-layer challenges the API rejects. Rate-limit and auth errors stay on the API path. A browser retry would not help.
Concurrent safety rests on distributed booking locks via Redis SET NX EX with UUID nonces and a Lua-script-safe release that atomically checks the nonce before deleting. A per-user circuit breaker implemented as a CLOSED → OPEN → HALF_OPEN state machine lives in Redis hashes so it survives worker restarts and coordinates across horizontally scaled instances. Detection evasion runs through a 26-profile browser identity pool. Each user is assigned a deterministic identity via DJB2 hash of their user ID, so the same user always looks like the same browser across API calls, TLS handshakes (via Impit), cookie jars, and viewport. Poll intervals use Poisson-distributed stagger offsets with time-of-day multipliers rather than fixed delays, which makes traffic statistically indistinguishable from organic use. Credentials are encrypted at rest with AES-256-GCM using per-record IVs and a separately stored GCM tag, decrypted only in worker memory at the moment of use. Stripe powers a four-tier billing system (Free, Pro, Elite, Day Pass) with transactional plan enforcement. The active-request count check runs inside the same DB transaction as the insert, so concurrent request creations cannot race past the cap.
Tradeoffs
The dual-strategy fallback adds 2–5 seconds of latency when the API breaks, but keeps the booking path alive during upstream changes that would otherwise drop the entire user base offline. Poisson-distributed polling is statistically cheaper to detect than fixed intervals and costs roughly 5× the clock-time variance. I accepted the variance because correlated bursts are what bot detectors watch for. Deterministic per-user browser identity consumes more cookie-jar storage than randomizing per-session would, and rules out opportunistic IP rotation; the lesson from detection research is that real users are consistent, not random, so consistency wins. The Redis-backed circuit breaker survives restarts at the cost of a Redis round-trip per booking attempt. A single-worker uvicorn equivalent was rejected early. Horizontal scaling was non-negotiable for release-window throughput, which is why every coordination primitive is Redis-backed rather than in-process.
Outcome
SeatWatch handled approximately fifty parallel user sessions across four Railway services plus managed Postgres 16 and Redis 7. The monitor queue processed thousands of availability checks daily with jittered scheduling; sniping mode landed first-poll attempts within milliseconds of the configured release timestamp, using a 500 ms early-fire offset to compensate for BullMQ scheduling latency. The codebase spans 419 TypeScript source files and roughly 64,400 lines, with 97 unit and integration test files plus 21 Playwright E2E specs. Ten Prisma models evolved across 16 migrations back the data layer. The Stripe billing system processed real transactions across all four plan tiers with no observed race condition or double-charge, and the distributed booking lock has yet to produce a double-book in production.
Learnings
The hardest problem was the distributed booking lock. My first implementation used a plain SET NX without a nonce, which worked until a slow worker’s lock expired, another instance acquired it, and the original worker deleted the new lock on completion: a textbook distributed-systems bug that I had to encounter firsthand before the fix (atomic nonce-checked deletion via a Lua script) made intuitive sense. The circuit breaker went through three iterations: first in-memory (lost state on restart), then Redis without a half-open probe (couldn’t recover cleanly), finally a proper state machine in Redis hashes. The broader lesson I now believe is that detection evasion is fundamentally about consistency, not randomness. A real user has a stable browser fingerprint, stable cookie jar, and stable request rhythm. Randomizing everything makes traffic look more suspicious, not less.