Scaling Real-Time Systems: From 1K to 1M Concurrent Users

A system serving a thousand concurrent users is a software problem; a system serving a million is an operations problem that happens to contain software. At 1K, a single well-tuned server cluster handles everything, latency is dominated by your code, and failures are rare enough to debug live. At 1M, the dominant costs are network, fan-out and state synchronization — and every component you trusted at small scale becomes a bottleneck or a single point of failure.

The uncomfortable truth is that most architectures do not scale continuously; they scale in phases separated by rewrites. The design that takes you from 1K to 50K is usually not the design that takes you from 50K to 1M, and pretending otherwise is how teams end up with a 'temporary' hack in the critical path two years later. We have taken client systems through both jumps, and the pattern is consistent: plan for the phase transition explicitly, and instrument for it long before you need it.

This article walks through the four architecture shifts that actually matter between these scales — connection management, state distribution, event flow, and failure containment — with the concrete thresholds where each one stops being optional.

Every real-time system lives or dies by its connection layer. WebSockets, long-polling or whatever transport you choose means each concurrent user holds an open file descriptor, memory for session state, and a path through your load balancers. At 1K users this is invisible. At 100K, a naive setup exhausts file descriptors and socket buffers on application servers long before CPU breaks. At 1M, the connection layer must be its own dedicated tier — purpose-built gateway nodes (we typically use dedicated WebSocket gateways or managed equivalents) that terminate connections, authenticate, and forward events to the core via an internal bus.

The key architectural move is separating the connection plane from the compute plane. Gateways should be dumb: they hold the socket, know which room or channel a user belongs to, and relay messages. All business logic runs behind them, horizontally scaled, reachable through the event bus. This separation is what lets you scale connections and compute independently — adding gateway nodes for more users without touching the services that process events.

Also budget for the thundering herd of reconnects. Mobile clients disconnect constantly — cell handoffs, app backgrounding, flaky Wi-Fi. A million-user system routinely sees 5-10% of its connections churn per minute. Exponential backoff with jitter on the client, and idempotent session resumption on the server, are not polish; they are load-bearing.

At small scale, shared state lives in one database and everyone is fine. At 1M concurrent users, the database is the first thing to die — not because it is weak, but because real-time workloads hammer it with read-heavy, low-latency demands that no single primary can serve. The fix is a layered state strategy: a cache tier (Redis or similar) holding hot session and presence state in front of the database, partitioned by user or room so no single shard carries a hot key, and the durable store behind everything handling writes asynchronously where possible.

Partitioning discipline matters more than raw hardware. A presence system for chat rooms should shard by room ID; a personal notifications system by user ID. The moment one celebrity room or viral channel concentrates a quarter of your traffic on one shard, your p99 latency collapses while your averages look healthy. Design the hot-key case explicitly: either split large rooms into sub-partitions or replicate their state read-locally across multiple cache nodes.

Equally important is deciding what state does not need to be global at all. Much real-time state is ephemeral — typing indicators, live cursors, transient presence — and should live in memory with graceful loss semantics rather than being persisted. Persisting the ephemeral is one of the most common ways real-time systems accidentally build their own bottleneck.

Below roughly 10K concurrent users, service-to-service calls are fine. Above that, synchronous call chains multiply latency and cascade failures: one slow downstream makes every upstream wait, and under load the whole graph degrades together. The shift is to make an event stream (Kafka, Kinesis, Pulsar or a managed equivalent) the backbone — producers emit events, consumers process them at their own pace, and no critical path depends on a synchronous round trip through five services.

Event streaming buys you three things at scale: decoupling (producers never block on consumers), buffering (the stream absorbs traffic spikes that would otherwise crash a synchronous chain), and replay (when a consumer bugs out, you reprocess from the log instead of losing events). For real-time features specifically, the pattern we use most is fan-out at the stream: one user action emits one event, and a fan-out consumer replicates it into per-room or per-user streams so delivery to a million subscribers is a parallel read problem, not a broadcast problem.

Backpressure is the concept that separates systems that survive spikes from ones that die in them. Every stage — gateway, stream, consumer, database write path — needs a defined behavior when it cannot keep up: drop the lowest-priority events, sample, queue with a bounded size, or shed load deliberately. An unbounded queue is not a safety net; it is a memory leak with a delay.

At 1K users, a bad deploy causes a ten-minute incident you remember clearly. At 1M, the same bad deploy touches a million sessions before anyone notices — unless observability is designed for scale. That means distributed tracing across the entire event path (from socket receipt to final delivery), saturation metrics per node and per shard, and alerting on percentiles rather than averages. Your p50 will look healthy while your p99 is already broken; at this scale, users experience the tail.

Failure containment follows the same logic as the architecture: blast radius must be bounded by design. Canary deploys roll out to a fraction of gateways first. Circuit breakers isolate slow dependencies. Region-level failover handles the day an availability zone hiccups. And every real-time feature ships with a degraded mode — if rich live updates fail, users fall back to polling or stale-plus-refresh rather than staring at a frozen screen. Degraded gracefully beats down completely, every single time.

Load testing closes the loop. You cannot discover 1M-scale behavior in production for the first time; simulate it. We run sustained-load tests at 2x projected peak, including churn (constant connect/disconnect), hot rooms, and injected downstream failures. The systems that pass those tests uneventfully are the ones whose launch day is boring — and boring is exactly what you want at a million concurrent users.