SysDesign Prep
← Problems

Notification System

mediummid / senior
Start timed practice

Send push, SMS and email notifications from many internal services reliably, with priorities, rate limits and user preferences.

Requirements

Functional

  • Internal services (orders, security, marketing) submit a notification for a user; the system delivers it over push (iOS/Android), SMS and email.
  • Users set preferences: which channels, which categories, quiet hours, and a global opt-out for marketing.
  • Priorities: transactional (OTP, fraud alert) must go out in seconds; marketing can take minutes and be batched.
  • Templates with per-locale content; the caller sends a template id and variables, not final text.
  • Delivery status is tracked (queued, sent, delivered, failed, bounced) and queryable per notification.

Non-functional

  • No transactional notification is ever lost: at-least-once delivery end to end.
  • No user receives the same notification twice because of a retry.
  • A slow or failing provider for one channel must not delay other channels.
  • Respect provider rate limits (e.g. SMS provider allows 1,000/s) and legal rules (no marketing SMS at 3 am).
  • p99 time from submit to provider hand-off < 5 s for transactional priority.

Out of scope

  • In-app notification inbox UI, A/B testing of content, campaign targeting and audience selection.

Back-of-envelope estimates

Assumptions

  • 50M users. Each gets ~5 notifications/day on average: 250M notifications/day.
  • Split: 60% push, 30% email, 10% SMS.
  • Marketing blasts are the peak: a campaign to 20M users in 10 minutes.

Throughput

  • Average: 250M / 86,400 s ≈ 2,900/s.
  • Campaign peak: 20M / 600 s ≈ 33K/s submitted. Queues absorb this; senders drain at provider rate limits (push providers accept tens of thousands/s; SMS maybe 1K/s so a 2M-recipient SMS blast takes ~35 minutes and that is fine).

Storage

  • Notification record (id, user, template, status, timestamps) ≈ 300 B. 250M/day × 300 B ≈ 75 GB/day. Keep 30 days hot ≈ 2.3 TB, then archive to object storage. Partition by day for cheap drops.
  • Dedup keys: 250M/day × ~60 B ≈ 15 GB/day in Redis with a 24-hour TTL. Fine on a small cluster.
  • Device tokens: 50M users × 2 devices × 200 B ≈ 20 GB. Small.

Cost sanity check

  • SMS at ~$0.01: 25M/day × $0.01 = $250K/day. SMS volume is the number to challenge in the interview; push and email are near free. This is why preferences and channel fallback matter.

API design

POST /v1/notifications

  • Body: { idempotencyKey, userId, templateId, variables, priority: "transactional" | "marketing", channels?: ["push","sms","email"], scheduleAt?, ttlSeconds? }
  • Returns 202 { notificationId, status: "queued" }. Returns the same notificationId if idempotencyKey was already seen.
  • Errors: 400 unknown template or missing variables, 404 unknown user, 429 caller over its submission quota.

GET /v1/notifications/{id}

  • Returns { status, perChannel: [{ channel, status, providerMessageId, attempts, lastError, updatedAt }] }.

PUT /v1/users/{userId}/preferences

  • Body: { channels: { push: true, sms: false, email: true }, categories: { marketing: false, security: true }, quietHours: { start: "22:00", end: "08:00", tz: "Europe/London" } }

Provider webhooks

  • POST /v1/webhooks/{provider}: delivery receipts, bounces, unsubscribes. Must be idempotent and verified by signature.

Notes

  • 202 not 201: the call returns before anything is sent. Callers poll or subscribe to status events.
  • ttlSeconds lets a caller say "an OTP older than 5 minutes is useless, drop it instead of retrying".

Data model

Entity: notification

  • notification_id (PK, ULID so it sorts by time), user_id, template_id, priority, category, variables (JSON), status, created_at, schedule_at, expires_at.

Entity: delivery (one per channel attempt stream)

  • delivery_id (PK), notification_id (FK, indexed), channel, provider, provider_message_id (indexed for webhooks), status, attempts, last_error, updated_at.

Entity: user_preference

  • user_id (PK), channel flags, category flags, quiet_hours, timezone, updated_at.

Entity: device_token

  • user_id, platform, token (PK), last_seen, valid.

Entity: template

  • template_id, locale, channel, subject, body, version. Immutable per version.

Dedup state

  • Redis SET idempotency_key → notification_id with a 24-hour TTL. Checked before anything is enqueued.

Storage choice

  • notification and delivery: a relational DB partitioned by day, or Cassandra keyed by notification_id. Access is by id and by user + time; both fit either store. Relational is easier for the status API and ops queries.
  • Preferences and device tokens: relational, small, read on every send; cache in Redis.
  • Templates: relational with a CDN-style in-memory cache in each worker.

High-level design

Submit path
1. A producer service calls POST /v1/notifications on the notification API.
2. The API validates the template and variables, checks the idempotency key in Redis, writes a notification row with status queued, and publishes a message to the intake queue. It returns 202.

Routing path
1. A router consumer reads from intake, loads user preferences and device tokens (cached), applies rules: channel opt-outs, category opt-outs, quiet hours (defer marketing until the window ends), de-duplication across channels, channel fallback (no valid push token → email).
2. It renders the template per channel and locale, writes one delivery row per channel, and publishes to the per-channel, per-priority queue: push-transactional, push-marketing, sms-transactional, and so on.

Send path
1. Channel workers consume their queue at a rate capped to the provider's limit, call the provider (APNs, FCM, Twilio, SES), and update the delivery row with the provider message id.
2. On failure: retry with backoff by re-publishing with a delay; after N attempts or past expires_at, move to a dead-letter queue and mark failed.

Feedback path

  • Providers call webhooks with delivered, bounced or unsubscribed. The webhook handler updates delivery status, invalidates dead device tokens, and flips preferences on unsubscribe.

Why separate intake, routing and sending?

  • Intake must be fast and never block on providers. Routing holds the business rules in one place. Senders are per-provider so a Twilio incident cannot back up push.

Deep dives

1. Exactly-once from the user's point of view: idempotency and dedup

  • Producer-side: the caller supplies idempotencyKey (e.g. "order-shipped:order123"). The API does SET NX in Redis; if the key exists, return the existing notification id. This stops producer retries creating duplicates.
  • Consumer-side: queues are at-least-once, so a worker can crash after calling Twilio but before acking. Before each provider call, the worker does SET NX on "sent:{delivery_id}" with a TTL; if it already exists, skip the call and ack. There is still a tiny window between the provider call and the SET; accept it or use provider-side idempotency keys where supported (SES and Twilio both accept one).
  • Order matters: write the dedup marker, then call the provider, then update the row. Explain why each order is wrong if swapped.

2. Retries, backoff and the dead-letter queue

  • Classify errors: 4xx from the provider (bad token, invalid number) is permanent → fail immediately and fix the data (invalidate the token). 5xx and timeouts are transient → retry.
  • Exponential backoff with jitter: delay = min(cap, base × 2^attempt) × random(0.5, 1.5). Base 2 s, cap 10 minutes, max 6 attempts for transactional. Jitter prevents synchronised retry storms after a provider outage.
  • Implement delays with a delay queue (SQS delay, RabbitMQ TTL + dead-letter exchange, or a Redis sorted set keyed by due time) rather than sleeping in the worker.
  • After max attempts or past expires_at: publish to a DLQ with the full payload and error history. Alert on DLQ depth. Provide a replay tool that re-enqueues DLQ messages after a provider recovers.

3. Queue topology, priorities and provider rate limits

  • One queue per channel × priority: 3 channels × 2 priorities = 6 queues. Transactional workers are always scaled; marketing workers scale on queue depth.
  • Isolation: a marketing SMS blast fills sms-marketing; sms-transactional stays empty and OTPs go out in seconds.
  • Provider rate limits: each worker pool holds a shared token bucket (in Redis) sized to the provider limit, e.g. 1,000/s for SMS. Workers take a token before each call; when the provider returns 429, back off that pool for the Retry-After period and shrink the bucket.
  • Provider failover: if the primary SMS provider errors above a threshold, a circuit breaker routes to a secondary provider. Keep provider_message_id namespaced by provider so webhooks still match.

4. Preferences, quiet hours and aggregation

  • Rules are evaluated once in the router, never in senders. Transactional and security notifications ignore marketing opt-out and quiet hours; marketing is deferred to the end of the quiet window in the user's timezone (publish with a delay, or store with schedule_at and let a scheduler enqueue it).
  • Aggregation: "Alice and 12 others liked your post" is produced by holding low-priority social notifications in a per-user buffer (Redis list with a 10-minute window) and flushing one summary. The buffer key is the user plus category.
  • Frequency caps: at most N marketing pushes per day per user, enforced with a daily counter in Redis.

5. Templates and localisation

  • Templates are versioned and immutable; a notification records the template version it rendered, so a later edit does not change history.
  • Rendering uses the user's locale with fallback (fr-CA → fr → en). Variables are validated against the template's declared schema at submit time so a missing variable fails fast with a 400 rather than sending a broken message.
  • Rendering happens in the router, once per channel, so senders are dumb and fast.

6. Observability and delivery status

  • Every state transition writes the delivery row and emits an event to a status topic. Dashboards show per-channel and per-provider sent, delivered, failed rates and p99 submit-to-sent latency.
  • Webhooks are the source of truth for delivered; without them, "sent" means "accepted by the provider". Expose both so callers do not confuse them.
  • Trace id from the producer flows through queue message attributes so a single notification can be followed end to end.

Trade-offs & wrap-up

Summary of decisions

  • At-least-once queues everywhere, with idempotency keys at submit and dedup markers before each provider call.
  • Three stages, intake → router → per-channel senders, so business rules live in one place and providers are isolated.
  • Per channel × priority queues with provider-sized token buckets and circuit breakers.
  • Exponential backoff with jitter, bounded attempts, TTLs for time-sensitive content, and a DLQ with replay.
  • Templates versioned and rendered once in the router.

What was given up

  • Strict exactly-once: a very small duplicate window remains between the provider call and the dedup marker.
  • Latency for marketing: it is deliberately deferred and batched.
  • Operational surface: six queues, several worker pools and webhook handlers to run.

With more time

  • Provider cost routing (cheapest SMS route per country), in-app inbox as a fourth channel, campaign-level throttling, and a per-user sending history that lets product teams reason about notification fatigue.