Chat App (WhatsApp / Messenger)

Functional

• One-to-one chat: send text (and small media references) to another user.
• Group chat: up to a few hundred members per group.
• Delivery receipts: sent (server has it), delivered (recipient device has it), read.
• Offline support: a recipient who is offline receives every message when they reconnect, in order.
• Message history: a device can fetch older messages for a conversation.
• Presence: online / last seen; typing indicators are nice-to-have.

Non-functional

• Low latency: a message between two online users arrives in under ~200 ms end to end.
• Ordering: within a conversation every participant sees the same order.
• Durability: once the server acknowledges a message it is never lost.
• Highly available: a connection server dying must not drop messages, only connections.
• Scale: hundreds of millions of daily active users, tens of billions of messages per day.

Out of scope

• Voice/video calls, stories, media transcoding, spam detection, end-to-end encryption key exchange (we discuss its impact in the deep dive).

Assumptions

• 500M DAU, each sends 40 messages/day → 20B messages/day.
• Average message ~100 bytes of text plus ~100 bytes of metadata (ids, timestamps, status) ≈ 200 B.
• Each user keeps one persistent connection most of the day; ~50% of DAU online at peak.

Traffic

• Writes: 20B / 86,400 s ≈ 230K messages/s average; peak 3× ≈ 700K/s.
• Each message produces at least one delivery push plus two receipt updates, so the fan-out write rate is 3–4× the ingest rate.
• Group messages multiply: a message to a 100-person group is 1 store write but 100 pushes.

Connections

• 250M concurrent WebSocket connections at peak.
• A tuned connection server holds ~200K–500K idle connections (limited by memory per socket, roughly 10–20 KB, and file descriptors). 250M / 250K ≈ 1,000 connection servers, plus headroom.

Storage

• 20B × 200 B = 4 TB/day of messages. Over 5 years ≈ 7 PB. Only feasible with a horizontally sharded wide-column store and a retention policy (e.g. drop server copies 30 days after delivery to all devices).
• Media is stored in object storage, referenced by URL; it dominates bytes but not the message path.

Memory for registry and presence

• Connection registry: 250M entries × ~50 B (user_id → server_id, timestamp) ≈ 12.5 GB. Fits a Redis cluster easily.
• Presence heartbeats every 30 s from 250M users ≈ 8M writes/s to presence. Batch and coalesce on the connection server, or drop to "last seen" granularity of minutes.

Bandwidth

• 700K msg/s × 200 B ≈ 140 MB/s ingest at peak. Small compared with media.

Connection

• WebSocket wss://chat.example.com/ws with an auth token on connect. The server returns a session id and resume cursor.
• Client sends a heartbeat (ping) every 30 s; the server closes idle sockets after 2 missed beats.

Over the socket (JSON or protobuf frames)

• send_message { client_msg_id, conversation_id, body, media_ref? } → server replies ack { client_msg_id, message_id, seq, server_ts }. client_msg_id makes retries idempotent.
• message (server → client) { message_id, conversation_id, seq, sender_id, body, server_ts }.
• receipt { message_id, conversation_id, status: delivered | read } in both directions.
• typing { conversation_id, is_typing } best effort, never persisted.

REST for everything that is not real time

• GET /v1/conversations?cursor= → list of conversations with last message and unread count.
• GET /v1/conversations/{id}/messages?before_seq=&limit=50 → history page, newest first.
• POST /v1/groups { name, member_ids[] } and POST /v1/groups/{id}/members.
• POST /v1/media/upload-url → pre-signed URL; the message carries the resulting media_ref.
• GET /v1/sync?since_cursor= → every message and receipt the device missed, used on reconnect.

Errors

• 401 bad token, 403 not a conversation member, 413 message too large, 429 sending too fast, 409 duplicate client_msg_id (returns the original ack).

Entity: message

• conversation_id (partition key), seq (clustering key, ascending)
• message_id (globally unique, Snowflake-style), sender_id, body, media_ref, server_ts
• Reads are always "messages for conversation X after/before seq N", which maps to a single partition range scan.

Entity: conversation

• conversation_id (PK), type (direct | group), created_at, last_seq
• conversation_member: (conversation_id, user_id) → joined_at, role, last_read_seq, muted

Entity: user_inbox (per-user pending deliveries)

• user_id (partition key), message_id, conversation_id, seq, enqueued_at
• Rows are removed once the device acks delivery. This is what offline sync reads.

Entity: receipt

• (message_id, user_id) → delivered_at, read_at. For groups, aggregate "delivered to all" lazily.

Storage choice

• Messages: a wide-column store (Cassandra, ScyllaDB, HBase) keyed by conversation_id with seq as the clustering column. Append-heavy, time-ordered reads, horizontal scale, cheap tiered storage for old partitions.
• Conversation metadata and membership: relational or a document store; small and relational-shaped.
• Connection registry and presence: Redis (user_id → connection_server_id, last_seen), with TTLs so dead servers age out.
• Hot conversations: the last ~50 messages per active conversation in a Redis list to serve history and reconnects without touching the wide-column store.

Components

• Connection servers: terminate WebSockets, hold millions of sockets, do nothing clever. Stateless apart from the sockets themselves.
• Connection registry (Redis): user_id → connection server id. Written on connect, deleted on disconnect, TTL as a safety net.
• Chat service: validates, assigns a per-conversation sequence number, persists the message, and routes it.
• Message store: wide-column, keyed by conversation.
• Inbox / delivery queue: per-user pending messages for offline recipients.
• Presence service: last seen and online status, fed by heartbeats.
• Media: object storage behind a CDN, uploaded directly by the client via pre-signed URLs.

Send path (Bob offline)

• Step 3 finds no registry entry. The chat service appends to Bob's inbox and, if enabled, hands off to the push notification system. When Bob reconnects, his client calls sync with its last cursor and drains the inbox in seq order.

Why a separate chat service rather than connection servers talking to each other directly?

• Sequence assignment and persistence need to happen exactly once per message; keeping that in one place keeps connection servers dumb and cheap to scale to thousands.

1. Connection servers and the registry

• Each server holds up to a few hundred thousand sockets. Clients connect through a layer-4 load balancer; once connected, the socket stays on that server, so there is no need for sticky HTTP sessions.
• On connect: SET user:{id} → server_id with a TTL of ~60 s, refreshed by heartbeats. On clean disconnect: DEL. If a server dies, its entries expire and clients reconnect elsewhere; the inbox catches any messages routed during the gap.
• Routing from the chat service to a specific server: either direct RPC to the server's address, or a pub/sub topic per server (e.g. Redis pub/sub or Kafka partition per server). Pub/sub is simpler to reason about when servers churn.
• A user with multiple devices has multiple registry entries; deliver to all, and each device acks separately.

2. Ordering with per-conversation sequence numbers

• Client timestamps cannot be trusted (clock skew, offline queueing). The server assigns a monotonically increasing seq per conversation.
• Implementation: route all writes for a conversation to one chat service shard (consistent hashing on conversation_id) and keep last_seq in memory, persisted with the message. Alternatively use an atomic INCR on a Redis key per conversation; losing Redis means re-reading max(seq) from the store before continuing.
• Clients keep last_seq per conversation. If a message arrives with seq > last_seq + 1, the client has a gap and fetches the missing range from history. This also makes "delivered in order" trivial: do not render seq N+2 until N+1 arrives.
• Trade-off: a single writer per conversation limits per-conversation throughput to one shard, which is fine; no single conversation needs more than a few thousand messages per second.

3. Receipts

• Sent: ack from the chat service after the store write succeeds. The client shows one tick.
• Delivered: the recipient device acks over its socket after it has persisted the message locally. For groups, store per-member delivery and show "delivered" once all members have it (or just the count).
• Read: emitted when the conversation is open on screen. Coalesce reads: one receipt with "read up to seq N" rather than one per message.
• Receipts are themselves tiny messages that take the same routing path back to the sender, but they are not sequenced or persisted with the same durability; losing a receipt is recoverable because the next sync resends the latest state.

4. Offline delivery and sync on reconnect

• For every message whose recipient has no live connection, append to user_inbox. Even for online recipients, write to the inbox first and delete after the device acks; this makes the push best-effort and the inbox the source of truth.
• On reconnect the client sends its cursor (last message_id or per-conversation last_seq). The server returns the inbox contents in order, then the client acks in batches and the server clears them.
• Mobile push: the inbox write also triggers a notification through APNs/FCM with no message body (or an encrypted body), so the user opens the app and the socket drains the inbox.
• Retention: the inbox for a user who never returns is capped (e.g. 30 days or 10K messages) and the rest is served from history on demand.

5. Group chat fan-out

• A group message is stored once under the group's conversation_id, then fanned out: one inbox write and one push per member. For a 256-member group that is 256 small writes, acceptable.
• Cap group size (WhatsApp historically 256, now 1,024). Above a few thousand members switch to pull: members fetch the conversation on open instead of receiving per-message pushes, like a channel.
• Membership changes must be sequenced too: "Alice added Bob at seq 1,042" so clients know which messages Bob may see.

6. Presence and typing

• Presence is heartbeat driven: every 30 s the connection server batches all its users into one presence update. Status is online if last heartbeat < 60 s ago, otherwise "last seen at".
• Only publish presence to users who are looking at you: a contact list open or a conversation open. Subscriptions live on the connection server and expire when the screen closes. This turns an N² problem into a modest fan-out.
• Typing indicators are fire-and-forget frames sent only to online members of the conversation, never persisted, rate limited to one per 3 s.

7. End-to-end encryption impact

• With E2EE (Signal protocol) the server stores ciphertext and cannot search, moderate, or render previews. Group keys are distributed per device, so adding a member means re-keying.
• Message routing, sequencing, receipts and inbox are unchanged; only the body is opaque. Multi-device requires the sender to encrypt once per recipient device, which grows the fan-out and is why linked devices are capped.

Summary of decisions

• WebSockets on dumb connection servers; a Redis registry maps user → server.
• A chat service assigns per-conversation sequence numbers and persists before acking.
• Wide-column message store keyed by (conversation_id, seq); per-user inbox for offline delivery.
• Inbox-first delivery makes the live push best effort and the inbox the source of truth.
• Receipts coalesced; presence subscription-based rather than broadcast.

What was given up

• A single sequencing shard per conversation caps per-conversation throughput (fine in practice).
• Server-side search and rich previews if E2EE is adopted.
• Exact "delivered to all" in large groups; we show counts and compute lazily.

With more time

• Media pipeline (thumbnails, transcoding), message search with client-side indexes, spam and abuse detection, multi-region active-active with conversation-home regions, and a proper key distribution service for E2EE.