Twilio System Design Scenarios

Component Design

1. API Gateway Layer

• Technology: AWS API Gateway + Lambda or ECS/Fargate
• Responsibilities:
  • Authentication (API key validation)
  • Rate limiting (per account)
  • Request validation
  • Idempotency key handling
  • Generate unique message SID
• Scaling: Multi-region deployment, auto-scaling based on request rate

2. Message Queue (Kafka)

• Why Kafka: High throughput, durability, ordering per partition
• Partitioning: By message_sid for ordering guarantees
• Topics:
  • messages.incoming - Newly accepted messages
  • messages.carrier-delivery - Ready to send to carrier
  • messages.status-updates - Delivery status changes
• Retention: 7 days for replay capability

3. Worker Fleet

• Technology: ECS/Kubernetes pods
• Consumer groups: Parallel processing, rebalancing on failures
• Responsibilities:
  • Consume from Kafka
  • Route to appropriate carrier based on destination
  • Handle carrier-specific protocols
  • Retry logic with exponential backoff
  • Publish status updates to Kafka
• Circuit breakers: Per carrier to handle failures

4. Storage (DynamoDB)

• Table: Messages
  • Partition key: account_id
  • Sort key: message_sid
  • Attributes: to, from, body, status, timestamps
  • GSI on message_sid for lookups
• Consistency: Eventual consistent reads for status checks
• Auto-scaling: On-demand capacity mode

5. Webhook Delivery

• Separate worker fleet consuming from messages.status-updates
• POST to customer webhook URL
• Retry with exponential backoff: 1min, 5min, 30min, 1hr, 6hr
• Circuit breaker per webhook URL
• Dead letter queue for failed webhooks

Design Approach

1. Data Classification

Different data has different consistency needs:

2. Routing Strategy

• Home region per account: Each account has a primary region (set at signup or based on location)
• Regional affinity: Route customer's API calls to their home region when possible
• Graceful degradation: If home region is down, route to nearest healthy region
• Data residency: EU accounts MUST have home region in EU (GDPR)

3. Handling Cross-Region Writes

When EU customer's request lands in US region:

1. Option A (Proxy): US region proxies request to EU region, waits for response
   • Pro: Strong consistency
   • Con: Higher latency (200ms+ penalty)
2. Option B (Local write + async reconcile): Accept in US, async replicate to EU
   • Pro: Low latency
   • Con: Potential conflicts, violates data residency
3. Recommended: Option A for control plane, local writes for data plane

4. Quota/Balance Enforcement

Challenge: Can't do synchronous cross-region checks (too slow)

5. Message History Queries

• Messages primarily queried in the region they were sent
• Each region maintains local message history
• Async replication to other regions (DynamoDB Global Tables or custom)
• If customer queries in non-home region: eventual consistency acceptable (might miss very recent messages)

6. Failure Scenarios

Scenario: EU region goes down

• DNS/routing layer detects failure, stops routing to EU
• EU customers routed to US region (nearest)
• US region acts as backup for EU accounts
• When EU recovers, catch up on replicated data, resume normal routing

Scenario: Network partition between regions

• Each region continues operating independently (AP in CAP)
• Quota enforcement becomes per-region (might over-deliver globally)
• When partition heals, reconcile quota usage
• If over-delivered, customer gets charged (better than losing messages)

This is YOUR wheelhouse! Align with your PayPal experience.

Design Principles

1. Cell Definition

A cell is:

• Fully isolated infrastructure stack (compute, storage, network)
• Separate VPC or namespace
• Independent deployment unit
• Handles subset of total traffic
• Fails independently without cascading

2. Cell Sizing Strategy

3. Routing Layer

• Mapping Service: Maps account_id → cell_id
• Storage: DynamoDB Global Table (highly available)
• Caching: Cached in API gateway for low latency
• Updates: Account growth triggers cell migration

4. Within Shared Cells: Additional Isolation

Even within shared cells, prevent noisy neighbors:

• Rate limiting: Per account
• Thread pool bulkheads: Per customer or per priority class
• CPU/Memory limits: Kubernetes resource quotas per customer namespace
• Connection pool limits: Per account to prevent DB exhaustion

5. Cell Migration

When customer outgrows shared cell:

1. Provision new dedicated cell
2. Dual-write messages to both old cell and new cell (shadowing)
3. Validate new cell working correctly
4. Update routing: new messages go to new cell only
5. Backfill historical data async if needed
6. Decommission old cell's resources for this customer

6. Operational Benefits

• Gradual rollout: Deploy change to Cell-Canary (synthetic traffic), then 1 shared cell, then all
• Feature flags per cell: Test new features on specific cells
• Blast radius: Bug in new deployment only affects one cell
• Capacity planning: Add cells when overall capacity reaches threshold

Identity Architecture

1. Developer Authentication

API Key Validation (Hot Path)

• Storage: DynamoDB
  • PK: api_key_sid
  • Attributes: account_id, permissions, created_at, last_used
• Caching: Redis/ElastiCache
  • Cache API key → account mapping for 5 minutes
  • 99% cache hit rate → sub-1ms auth check
  • Cache miss → query DynamoDB (5-10ms)
• Key rotation: Support multiple active keys, graceful deprecation

Console Login (OAuth2 / OIDC)

• Use Auth0, Cognito, or custom OAuth2 provider
• Support username/password + MFA
• Issue JWT tokens for session management
• Short-lived access tokens (15 min), long-lived refresh tokens

Enterprise SSO

• SAML 2.0 integration for enterprise customers
• Customer configures their IdP (Okta, Azure AD)
• Twilio acts as service provider (SP)
• JIT (Just-In-Time) provisioning of accounts

2. Service-to-Service Authentication

Why it matters

When API Gateway calls Billing Service, how do we authenticate?

Approach: Service Accounts + JWTs

• Each service has a service account with credentials
• Services request short-lived JWT tokens from auth service
• JWT includes: service_id, permissions, expiry
• Target service validates JWT (using public key or shared secret)
• Token expires in 1 hour, auto-renewed

Alternative: mTLS (Mutual TLS)

• Each service has X.509 certificate
• TLS handshake validates both client and server
• More secure but more operational overhead
• Good for sensitive services (billing, identity)

3. Account Hierarchy

Twilio supports parent accounts with sub-accounts:

Data Model

• Accounts table:
  • account_sid (PK), parent_account_sid, name, status
• API Keys table:
  • api_key_sid (PK), account_sid, permissions_scope
• Query pattern: Given API key, lookup account, check if account is active, check permissions

4. Audit Logging

• Every auth event logged: API key used, token issued, login attempt
• Log to Kafka → S3 for compliance
• ElasticSearch for searchable audit trail
• Include: timestamp, account_id, action, ip_address, result

5. Rate Limiting & Abuse Prevention

• Rate limit failed login attempts per IP: 10 per minute
• Rate limit API key validation per key: 10K per second
• CAPTCHA after 3 failed logins
• Temporary account lockout after 10 failed attempts

Click "Show Solution" to reveal the approach. Try designing it yourself first!

Recommended Approach: Redis with Sliding Window

Architecture

Redis Data Structure: Sorted Set

For each account, store timestamps of recent requests:

• Key: rate_limit:{account_id}:{window}
• Value: Sorted set with score = timestamp
• Example: rate_limit:AC123:minute → {1700000001, 1700000002, ...}

Algorithm: Sliding Window Counter

function checkRateLimit(account_id, limit, window_seconds):
    current_time = now()
    window_start = current_time - window_seconds

    # 1. Remove entries older than window
    ZREMRANGEBYSCORE rate_limit:{account_id} -inf window_start

    # 2. Count remaining entries
    count = ZCARD rate_limit:{account_id}

    # 3. Check if under limit
    if count < limit:
        # 4. Add current request
        ZADD rate_limit:{account_id} current_time current_time

        # 5. Set TTL to auto-expire old data
        EXPIRE rate_limit:{account_id} window_seconds

        return ALLOWED, remaining = (limit - count - 1)
    else:
        return DENIED, remaining = 0

Making it Atomic: Lua Script

Run all Redis commands in a single Lua script for atomicity:

local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local current_time = tonumber(ARGV[3])
local window_start = current_time - window

redis.call('ZREMRANGEBYSCORE', key, '-inf', window_start)
local count = redis.call('ZCARD', key)

if count < limit then
    redis.call('ZADD', key, current_time, current_time)
    redis.call('EXPIRE', key, window)
    return {1, limit - count - 1}  -- allowed, remaining
else
    return {0, 0}  -- denied, remaining
end

Response Headers

Include in every API response:

• X-RateLimit-Limit: 1000 - Total limit
• X-RateLimit-Remaining: 742 - Requests remaining
• X-RateLimit-Reset: 1700000060 - When window resets

Handling Multiple Time Windows

Support multiple limits simultaneously:

• 1000 requests per minute
• 10,000 requests per hour
• 100,000 requests per day

Solution: Check each limit independently, deny if ANY limit exceeded

Scaling Redis

• Redis Cluster: Partition by account_id (consistent hashing)
• Replication: Redis replica for read availability
• Fallback: If Redis is down, fail open or use local in-memory limit (less accurate)

Alternative Approaches