CI/CD Pipelines

What is CI/CD?

Continuous Integration (CI)

Automatically build and test code changes frequently (multiple times per day).

• Developers commit code to shared repository (Git)
• Automated build triggered on every commit
• Run automated tests
• Fast feedback (pass/fail within minutes)
• Catches integration issues early

Continuous Delivery (CD)

Code is always in deployable state. Manual approval for production.

• Extends CI with deployment automation
• Deploy to staging automatically
• Manual gate before production
• Reduces deployment risk

Continuous Deployment (CD)

Fully automated - every change goes to production automatically.

• No manual intervention
• All tests pass → deploy to production
• Requires high confidence in tests
• Fastest time to market

Benefits of CI/CD

• Faster feedback: Bugs found within minutes, not days
• Reduced risk: Small, frequent changes easier to debug
• Higher quality: Automated testing catches regressions
• Faster time to market: Deploy features quickly
• Better collaboration: Integration issues surface early
• Reduced manual work: Automation eliminates repetitive tasks
• Rollback capability: Easy to revert bad changes

Typical CI/CD Pipeline

┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   Commit    │───>│    Build    │───>│    Test     │───>│   Package   │───>│   Deploy    │
│  (Git Push) │    │  (Compile)  │    │(Unit, Integ)│    │  (Docker)   │    │  (Staging)  │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
                                                                                      │
                                                                                      v
                                                                           ┌─────────────────┐
                                                                           │Manual Approval  │
                                                                           │ (for prod)      │
                                                                           └─────────────────┘
                                                                                      │
                                                                                      v
                                                                           ┌─────────────────┐
                                                                           │Deploy Production│
                                                                           └─────────────────┘

Stage 1: Source / Trigger

• Code commit triggers pipeline
• Pull request triggers checks
• Scheduled builds (nightly)
• Manual trigger

Stage 2: Build

• Checkout code from Git
• Install dependencies
• Compile code (if compiled language)
• Run linters, static analysis
• Build artifacts (JAR, binary, etc.)

name: Build
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Lint
        run: npm run lint

      - name: Build
        run: npm run build

Stage 3: Test

• Unit tests: Fast, isolated tests
• Integration tests: Test component interactions
• E2E tests: Critical user journeys (slow, run selectively)
• Security scans: Vulnerability scanning (Snyk, Trivy)
• Code coverage: Ensure adequate test coverage

- name: Run unit tests
  run: npm test -- --coverage

- name: Run integration tests
  run: npm run test:integration

- name: Security scan
  run: npm audit

- name: Upload coverage
  uses: codecov/codecov-action@v3

Stage 4: Package / Artifact Creation

• Build Docker image
• Create deployment package
• Version artifacts (semantic versioning)
• Push to artifact repository

- name: Build Docker image
  run: |
    docker build -t myapp:${{ github.sha }} .
    docker tag myapp:${{ github.sha }} myapp:latest

- name: Push to registry
  run: |
    echo ${{ secrets.DOCKER_PASSWORD }} | docker login -u ${{ secrets.DOCKER_USERNAME }} --password-stdin
    docker push myapp:${{ github.sha }}
    docker push myapp:latest

Stage 5: Deploy

• Deploy to staging environment
• Run smoke tests
• Manual approval gate (for production)
• Deploy to production
• Health checks

Build Optimization

1. Caching Dependencies

# GitHub Actions - Cache node_modules
- uses: actions/cache@v3
  with:
    path: ~/.npm
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-node-

# GitLab CI - Cache
cache:
  paths:
    - node_modules/
    - .npm/

2. Parallel Jobs

# Run tests in parallel
jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - run: npm test

  integration-tests:
    runs-on: ubuntu-latest
    steps:
      - run: npm run test:integration

  lint:
    runs-on: ubuntu-latest
    steps:
      - run: npm run lint

# All run simultaneously

3. Incremental Builds

• Only rebuild changed components (monorepos)
• Use build tools that support incremental compilation (Gradle, Bazel)
• Skip tests for unchanged code

4. Docker Layer Caching

# Use BuildKit for better caching
DOCKER_BUILDKIT=1 docker build .

# Use registry as cache
docker build --cache-from myapp:latest -t myapp:new .

1. Blue-Green Deployment

Two identical environments. Switch traffic from old (blue) to new (green).

Before Deployment:
┌────────────┐
│   Users    │
└─────┬──────┘
      │
      v
┌─────────────┐      ┌─────────────┐
│ Load        │      │             │
│ Balancer    │      │             │
└──────┬──────┘      │             │
       │             │             │
       v             │             │
┌─────────────┐      │             │
│ Blue (v1.0) │      │ Green (idle)│
│ ████████    │      │             │
└─────────────┘      └─────────────┘

After Deployment (instant switch):
┌────────────┐
│   Users    │
└─────┬──────┘
      │
      v
┌─────────────┐      ┌─────────────┐
│ Load        │      │             │
│ Balancer    │      │             │
└──────┬──────┘      │             │
       │             │             │
       v             v             │
┌─────────────┐      ┌─────────────┐
│ Blue (v1.0) │      │ Green (v2.0)│
│ (idle)      │      │ ████████    │
└─────────────┘      └─────────────┘

2. Canary Deployment

Roll out to small subset of users first. Gradually increase traffic if healthy.

Phase 1: 5% traffic to canary
┌────────────┐
│   Users    │
└─────┬──────┘
      │
      v
┌─────────────┐
│ Load        │ 95% ──> v1.0 (stable)
│ Balancer    │  5% ──> v2.0 (canary)
└─────────────┘

Phase 2: 50% traffic (if healthy)
┌─────────────┐
│ Load        │ 50% ──> v1.0
│ Balancer    │ 50% ──> v2.0
└─────────────┘

Phase 3: 100% traffic
┌─────────────┐
│ Load        │ 100% ─> v2.0
│ Balancer    │
└─────────────┘

# Kubernetes Canary with Flagger
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: myapp
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  service:
    port: 8080
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
  canaryAnalysis:
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
    - name: request-duration
      thresholdRange:
        max: 500

3. Rolling Deployment

Gradually replace instances one-by-one or in batches.

Step 1: Replace 1 instance
v1: [██] [██] [██] [  ]
v2: [  ] [  ] [  ] [██]

Step 2: Replace another
v1: [██] [██] [  ] [  ]
v2: [  ] [  ] [██] [██]

Step 3: Replace all
v1: [  ] [  ] [  ] [  ]
v2: [██] [██] [██] [██]

# Kubernetes RollingUpdate
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2        # Create 2 extra pods during update
      maxUnavailable: 1  # Max 1 pod unavailable at a time

4. Feature Flags / Dark Launch

Deploy code to production but keep features disabled. Enable for specific users.

// Feature flag example
if (featureFlags.isEnabled('newCheckout', user.id)) {
  return ;
} else {
  return ;
}

// Gradually increase percentage
featureFlags.setRolloutPercentage('newCheckout', 10);  // 10% of users

What is GitOps?

Use Git as single source of truth for infrastructure and application config. Changes to Git automatically deployed.

Principles

1. Declarative: Desired state described in Git (YAML, Terraform)
2. Versioned & Immutable: Git history is audit trail
3. Pulled Automatically: Agent pulls from Git, applies changes
4. Continuously Reconciled: Actual state matches desired state

Traditional Push-based CD:
┌──────────┐       ┌──────────┐       ┌──────────┐
│   Git    │──────>│   CI     │──────>│  Cluster │
│  Commit  │       │ Pipeline │       │  (push)  │
└──────────┘       └──────────┘       └──────────┘

GitOps Pull-based:
┌──────────┐       ┌──────────┐       ┌──────────┐
│   Git    │<──────│  Agent   │<──────│  Cluster │
│  Commit  │       │ (ArgoCD) │       │  (pull)  │
└──────────┘       └──────────┘       └──────────┘
                   Polls Git every 3min,
                   applies if changed

GitOps Tools

Example: Argo CD

# Application definition
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  source:
    repoURL: https://github.com/org/repo
    targetRevision: main
    path: k8s/manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true      # Delete resources not in Git
      selfHeal: true   # Auto-sync if drift detected