BizComms Overview

A production-grade, multi-region alert notification system built on AWS with cell-based architecture, demonstrating enterprise patterns for failure domain isolation, strong data consistency, and automated operations.

What is BizComms?

BizComms is a real-time alert notification system that monitors HackerNews for new articles and sends SMS and voice notifications to subscribers via Twilio. It's designed to demonstrate:

Key Features

System Architecture

High-Level Architecture

Cell-Based Architecture

Each region operates as an independent cell with complete isolation. This design limits the blast radius of failures and enables independent scaling.

Cell Components

Network Architecture

Subnet Design

• Public Subnets: NAT Gateways, Application Load Balancer
• Private Subnets: EKS nodes and pods (no direct internet access)
• Database Subnets: Aurora clusters (isolated from internet)

Infrastructure Components

AWS Services Used

Terraform Module Structure

Resource Sizing

Microservices

Service Architecture

1. API Service

Endpoints

Configuration

# Kubernetes Deployment
replicas: 3
resources:
  requests:
    memory: 256Mi
    cpu: 250m
  limits:
    memory: 512Mi
    cpu: 500m

# Auto-scaling
minReplicas: 2
maxReplicas: 10
targetCPU: 70%
targetMemory: 80%

2. Worker Service

Processing Flow

Error Handling

• Retry Logic: SQS visibility timeout (30s) with max receive count (3)
• Dead Letter Queue: Failed messages move to DLQ after 3 attempts
• Idempotency: Message deduplication using message_id

3. Poller Service

Polling Logic

# Every 5 minutes:
1. Fetch top 30 stories from HackerNews API
2. Compare with previous state (stored in DynamoDB)
3. For each new story:
   - Fetch story details
   - POST to /v1/alerts endpoint
   - Update state in DynamoDB
4. Sleep until next poll interval

Data Flow & Patterns

End-to-End Alert Flow

Data Consistency Model

Strong Consistency (Aurora)

Used for transactional data requiring ACID guarantees:

• Alerts table: Alert metadata, status, created_at
• Recipients table: Phone numbers, preferences, active status
• Replication: Aurora Global Database (<1 second lag)

Eventual Consistency (DynamoDB)

Used for high-volume, time-series data:

• Delivery status: Twilio SID, status, timestamps
• Replication: DynamoDB Global Tables (typically <1 second)
• TTL: Auto-delete records after 30 days

Messaging Patterns

1. Fan-Out Pattern (SNS → SQS)

2. At-Least-Once Delivery

SQS guarantees at-least-once delivery with these mechanisms:

• Visibility Timeout: 30 seconds (message hidden during processing)
• Max Receive Count: 3 attempts before moving to DLQ
• Message Retention: 14 days
• Dead Letter Queue: Failed messages for investigation

3. Idempotency

# Worker ensures idempotent processing
message_id = message.get("MessageId")

# Check if already processed (in DynamoDB)
if delivery_status_exists(message_id):
    logger.info(f"Message {message_id} already processed")
    return

# Process and store status atomically
process_alert(alert_id)
store_delivery_status(message_id, status)

Regional Failover

Failover Process:

1. Route53 health check detects primary region failure
2. DNS automatically routes traffic to secondary region
3. Aurora Global Database promotes secondary to primary (manual)
4. Applications in secondary region continue serving requests

API Reference

Authentication

curl -H "X-API-Key: your-api-key-here" \
  https://api.bizcomms.example.com/v1/alerts

Create Alert

POST /v1/alerts

Request Body

{
  "title": "New HackerNews Story",
  "message": "Check out this awesome article!",
  "priority": "high",
  "delivery_method": "sms",
  "recipient_filters": {
    "tags": ["tech", "news"],
    "active_only": true
  }
}

Response (202 Accepted)

{
  "alert_id": "alert_abc123",
  "status": "queued",
  "created_at": "2025-01-01T12:00:00Z",
  "estimated_recipients": 42
}

Get Alert Status

GET /v1/alerts/{id}/status

Response (200 OK)

{
  "alert_id": "alert_abc123",
  "total_recipients": 42,
  "delivery_status": {
    "sent": 40,
    "delivered": 38,
    "failed": 2,
    "pending": 0
  },
  "deliveries": [
    {
      "recipient_id": "recip_123",
      "phone_number": "+1234567890",
      "status": "delivered",
      "twilio_sid": "SM1234567890abcdef",
      "sent_at": "2025-01-01T12:00:05Z",
      "delivered_at": "2025-01-01T12:00:07Z"
    }
  ]
}

Create Recipient

POST /v1/recipients

Request Body

{
  "name": "John Doe",
  "phone_number": "+1234567890",
  "preferences": {
    "sms_enabled": true,
    "voice_enabled": false
  },
  "tags": ["tech", "news"],
  "active": true
}

Response (201 Created)

{
  "recipient_id": "recip_123",
  "name": "John Doe",
  "phone_number": "+1234567890",
  "created_at": "2025-01-01T12:00:00Z"
}

Error Responses

Deployment Guide

Prerequisites

• AWS Account with Administrator access
• AWS CLI configured
• Terraform 1.6+
• kubectl 1.28+
• Helm 3.13+
• Docker 24.0+

Deployment Steps

Step 1: Terraform Backend Setup

cd terraform/backend-setup
terraform init
terraform apply

# Output: S3 bucket and DynamoDB table for state

Step 2: Deploy Primary Region (us-east-1)

cd ../environments/us-east-1

# Initialize Terraform
terraform init

# Review plan
terraform plan

# Apply infrastructure
terraform apply

# Capture outputs
terraform output -json > outputs.json

Step 3: Deploy Secondary Region (us-west-1)

cd ../us-west-1

terraform init
terraform plan
terraform apply

terraform output -json > outputs.json

Step 4: Build and Push Docker Images

# Authenticate to ECR
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com

# Build images
docker build -f docker/Dockerfile.api -t bizcomms-api:latest .
docker build -f docker/Dockerfile.worker -t bizcomms-worker:latest .
docker build -f docker/Dockerfile.poller -t bizcomms-poller:latest .

# Tag and push
docker tag bizcomms-api:latest ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/bizcomms-api:latest
docker push ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/bizcomms-api:latest

# Repeat for worker and poller

Step 5: Deploy Kubernetes Applications

# Configure kubectl
aws eks update-kubeconfig --region us-east-1 --name bizcomms-cell-use1-01-eks

# Get WAF ARN from Terraform
WAF_ARN=$(cd terraform/environments/us-east-1 && terraform output -raw waf_web_acl_arn)

# Deploy with Helm
helm install bizcomms ./helm/bizcomms \
  -f helm/bizcomms/values-us-east-1.yaml \
  --set ingress.wafAclArn="$WAF_ARN" \
  --namespace bizcomms \
  --create-namespace

# Verify deployment
kubectl get pods -n bizcomms
kubectl get ingress -n bizcomms

Step 6: Update Route53

# Get ALB DNS name
ALB_DNS=$(kubectl get ingress bizcomms-ingress -n bizcomms -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

# Update Terraform variables
cd terraform/environments/us-east-1
echo "alb_dns_name = \"$ALB_DNS\"" >> terraform.tfvars

# Re-apply to update Route53
terraform apply -target=module.route53

Deployment Architecture

Verification

# Check API health
curl https://api.bizcomms.example.com/health

# Create test alert
curl -X POST https://api.bizcomms.example.com/v1/alerts \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Test Alert",
    "message": "Hello from BizComms!",
    "priority": "high",
    "delivery_method": "sms"
  }'

# Check logs
kubectl logs -n bizcomms -l app=bizcomms-api --tail=100
kubectl logs -n bizcomms -l app=bizcomms-worker --tail=100

Security

Security Architecture

Security Best Practices Implemented

1. No Hardcoded Credentials

• IRSA: Pods assume IAM roles via OIDC provider
• Secrets Manager: Twilio credentials, DB passwords, API keys
• Environment Variables: Only non-sensitive config

2. Least Privilege IAM

// API Pod Role
{
  "Effect": "Allow",
  "Action": [
    "secretsmanager:GetSecretValue",
    "sns:Publish"
  ],
  "Resource": [
    "arn:aws:secretsmanager:*:*:secret:bizcomms-*",
    "arn:aws:sns:*:*:bizcomms-*"
  ]
}

3. Network Isolation

4. Encryption

• In Transit: TLS 1.2+ for all API calls, Aurora connections
• At Rest: KMS encryption for Aurora, DynamoDB, S3
• Secrets: KMS-encrypted in Secrets Manager

5. WAF Protection

# Managed Rules Applied
- AWSManagedRulesSQLiRuleSet    # SQL injection
- AWSManagedRulesKnownBadInputsRuleSet  # Known threats
- AWSManagedRulesCommonRuleSet  # OWASP Top 10

# Rate Limiting
- 1000 requests per 5 minutes per IP
- Custom response: 429 Too Many Requests

6. Audit Logging

• CloudTrail: All AWS API calls logged
• CloudWatch Logs: Application logs (API, Worker, Poller)
• VPC Flow Logs: Network traffic analysis

Security Checklist

Monitoring & Observability

Observability Stack

CloudWatch Dashboards

Automated dashboards created by Terraform monitoring module:

1. API Service Dashboard

• Request count (per endpoint)
• Latency (p50, p95, p99)
• Error rate (4xx, 5xx)
• Active connections

2. Worker Service Dashboard

• SQS queue depth
• Messages processed/minute
• Processing duration
• Twilio delivery success rate

3. Infrastructure Dashboard

• Aurora: CPU, connections, IOPS, replication lag
• DynamoDB: Read/write capacity, throttles
• EKS: Pod count, node utilization

CloudWatch Alarms

Log Aggregation

# Query API logs for errors
aws logs tail /aws/eks/bizcomms-cell-use1-01/api --follow \
  --filter-pattern "ERROR"

# CloudWatch Insights query
fields @timestamp, @message
| filter @message like /5xx/
| stats count() by bin(5m)

Health Checks

Kubernetes Probes

# Liveness Probe
livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 10

# Readiness Probe
readinessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 10
  periodSeconds: 5

Route53 Health Checks

• Endpoint: https://api.bizcomms.example.com/health
• Interval: 30 seconds
• Failure threshold: 3 consecutive failures
• Action: Failover to secondary region

Cost Optimization

Estimated Monthly Costs (Production)

Cost Optimization Strategies

1. Compute Optimization

# Karpenter Provisioner
spec:
  requirements:
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["spot", "on-demand"]
  limits:
    resources:
      cpu: 1000
      memory: 1000Gi

2. Database Optimization

• Aurora Serverless v2: Auto-scales based on load (future consideration)
• Read Replicas: Add read replicas only when needed
• Right-Sizing: Start with db.t4g.medium for dev/staging

3. Storage Optimization

• DynamoDB On-Demand: Pay only for actual reads/writes (no provisioned capacity waste)
• TTL: Auto-delete delivery status after 30 days
• S3 Lifecycle: Move CloudWatch logs to Glacier after 90 days

4. Network Optimization

5. Monitoring Optimization

• Log Retention: 7 days for debug logs, 30 days for error logs
• Metric Filters: Only send necessary metrics to CloudWatch
• Sampling: Sample debug logs (10%) instead of 100%

Cost by Environment