High-Scale Real-Time Notification Architecture with NestJS, Redis Pub/Sub, and WebSocket

1. Why Real-Time Notifications Frequently Become Your System's Biggest Point of Failure

Have you ever endured that heart-stopping moment where an apparently straightforward feature—like instant order status alerts or chat notifications—suddenly brings your entire production backend to its knees during a major promotional launch?

That exact ordeal is what my engineering team and I encountered when we first rolled out a live order tracking feature. In our quiet staging environments, everything had operated smoothly. But the moment we deployed to production with tens of thousands of active users, our naive architectural shortcuts ignited a cascade of critical failures:

• Database Connection Poolℹ Exhaustion: Mobile and web clients were configured to poll the order status endpoint every 2 seconds. With 50,000 active shoppers, our backend was battered by 25,000 requests per secondℹ merely checking for status changes that had not yet occurred! Primary database CPU spiked to 98%, stalling checkout transactions across the platform.
• The "Client Dual-Hitℹ" Race Conditionℹ Disaster: To reduce pollingℹ load, we initially tried a quick shortcut: frontends fired a REST API mutation, and then manually triggered a client-side socket emit. On spotty cellular networks, the HTTP call cleared but the socket emit dropped. Users had their credit cards charged without ever receiving confirmation banners, sparking panic in our customer support queues.
• TCP Memory Isolation in Multi-Podℹ Clusters: When our WebSocket gateway scaled across 10 Kubernetes podsℹ, users connected to Podℹ A never received live notifications when the order was fulfilled on Podℹ B!

From these incidents, we formulated an unshakeable architectural rule: never entrust real-time distributed orchestration to client devices, and never depend on a single server node's local memory.

2. Notification Topology & Architectural Comparison Matrix

Before walking through the code implementation, let us examine how traditional naive approaches compare to our decoupled push architecture:

3. Three Critical Problems Behind Traditional Notification Failures

Problem 1: The Short Polling Trap Draining Connection Pools

When client applications poll on a 2-second interval:

• 50,000 active users generate 25,000 requests per secondℹ (RPSℹ) of redundantℹ database lookups.
• Primary PostgreSQL connection pools are drained (connection starvationℹ), query latencies jump from 15ms to 12,000ms, and paying customers encounter 500 error screens.

Solution 1: Shifting from Pull to Event-Driven Push

By shifting to persistent WebSockets bridged via Redis, the 25,000 RPSℹ pollingℹ storm drops immediately to 0 RPSℹ. TCP connections stay open with negligible overhead, and data flows only when real business events happen.

Problem 2: The Client Dual-Hit Anti-Pattern

When developers attempt a quick workaround:

If a mobile user traverses a subway tunnel between line 1 and line 2, the HTTP order clears, but the socket emit is lost forever. Furthermore, malicious actors can reverse-engineer the client script to spam fake notification events into other users' socket rooms.

Solution 2: Single-Hit Stateless REST Ingestion

Clients issue a single HTTP request. The backend securely authorizes the session, writes the transaction to PostgreSQL, and internally publishes the event to the distributed notification bus.

Problem 3: TCP Connection Memory Isolation Across Kubernetes Pods

In a containerized environment:

• User A connects to WebSocket Gateway on Podℹ 1.
• User B connects to WebSocket Gateway on Podℹ 2.
• User A triggers an action handled by API Podℹ 3. If Podℹ 3 attempts to broadcast the event solely using its local Node.js memory, User B on Podℹ 2 will never receive the alert because User B's TCP socket lives in Podℹ 2's process memory.

Solution 3: Redis Pub/Sub as the Distributed Event Bus

Redis acts as the shared nervous system across all podsℹ. Podℹ 3 publishes one message to Redis, and all WebSocket podsℹ simultaneously receive the event and push it out to their locally connected sockets.

4. The Flash Sale Incident: When 120,000 Concurrent Users Melted Our Database

Last quarter, our platform launched a flash sale. Within the first five minutes, 120,000 eager shoppers rushed into the catalog.

Because our legacy codebase still relied on a mixture of pollingℹ and client socket emits:

1. 1Saturated mobile networks caused 14% of client socket requests to timeout. Thousands of shoppers assumed their checkout had failed, despite money being deducted from their balances.
2. 2Panic-driven page refreshes ignited a 45,000 RPSℹ pollingℹ stampedeℹ that overwhelmed our PostgreSQL cluster.
3. 3API and socket podsℹ starved each other of CPU until the entire infrastructure crashed for 25 minutes.

That catastrophic incident forced us to completely reconstruct our notification pipeline around the decoupled NestJS and Redis Pub/Subℹ architecture detailed below.

5. Production NestJS Implementation (TypeScript)

Here is the modular, type-safe architecture we deployed to production across our Kubernetes clusters:

1. Dedicated Redis Pub/Sub Service

[!TIP]

The ioredis Connection Rule:

Once an ioredis client issues SUBSCRIBE, that connection is strictly locked in listening mode and rejects write commands like PUBLISH or SET. You must maintain dedicated instances for publisherClient and subscriberClient.

2. Order Controller & Service (Stateless REST Ingestion)

Clients trigger this single HTTP endpoint without managing any socket connections from the frontend:

3. NestJS WebSocket Gateway (Distributed Pod Broadcaster)

6. Real Production Metrics: Before vs After

Following our migration to this Decoupled Push architecture in production, our metrics told a dramatic story:

7. 4 Core Engineering Lessons from the Trenches

1. 1Client Single-Hit Principle: Never burden client applications with distributed notification orchestration. Clients execute one standard HTTP mutation, and the backend handles event broadcasting.
2. 2Decouple Statelessℹ APIs from Statefulℹ Sockets: Separate REST API podsℹ from WebSocket Gateway podsℹ. REST podsℹ can be freely redeployed or autoscaled without severing tens of thousands of active socket sessions.
3. 3Always Isolate ioredis Publisher and Subscriber Instances: Because Redis protocol locks connections in listening mode during SUBSCRIBE, maintaining dedicated client instances avoids silent write failures.
4. 4Transient vs Persistent Storage Balance: Utilize Redis Pub/Subℹ for lightweight, memory-efficient real-time delivery, but always store notification histories in your primary database so offline users never miss important updates.