Zero-Downtime Deployments Without the Drama

TL;DR

#

• Zero-downtime deployments aren’t just for FAANG. Small teams can do it with basic patterns.
• Blue-green works great for stateless apps. Rolling updates handle everything else.
• Database migrations are the real villain. Use expand-contract — and budget three deploys, not two.
• Readiness probes must be shallow. A probe that checks your database will take every replica out of rotation at the same moment.
• Start with one service. Perfect the process. Then expand.

Who this is for

#

Teams shipping to production multiple times per week who are tired of “scheduled maintenance” windows. You have 3-10 engineers, basic CI/CD in place, and customers who notice when you deploy at 3pm on a Tuesday.

The deployment patterns that actually matter

#

Forget the 47 different deployment strategies you read about. For small teams, two patterns cover 95% of use cases.

Blue-green deployments

#

Perfect for stateless applications. You run two identical environments (blue and green). Deploy to the inactive one, test it, then switch traffic.

The trick with nginx is to keep the switch in exactly one place — a file with a single line in it that your deploy script rewrites:

# /etc/nginx/conf.d/active-upstream.conf
# This entire file is what the deploy script swaps.
upstream app_current {
    server 10.0.1.10:3000;   # blue
}

# /etc/nginx/sites-available/app
include /etc/nginx/conf.d/active-upstream.conf;

server {
    listen 80;
    server_name app.example.com;

    location / {
        proxy_pass http://app_current;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Switching is then two lines in your deploy script:

printf 'upstream app_current {\n    server 10.0.1.20:3000;   # green\n}\n' \
  > /etc/nginx/conf.d/active-upstream.conf
nginx -t && nginx -s reload

The reload itself costs you nothing. nginx -s reload is graceful: the master process starts new workers with the new config, keeps the listening sockets open, and lets old workers finish their in-flight requests before exiting. No connection is dropped and no request is refused. Don’t budget milliseconds of downtime for it — budget zero, and spend your attention on the application’s own shutdown behaviour instead, which is where requests actually get lost.

Rolling updates

#

Better for containerized workloads or when you can’t afford double infrastructure. Replace instances one at a time.

If you’re on Kubernetes (even k3s), this is built-in:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  # selector is required, and it must match the pod template labels below.
  # Without it the API server rejects the manifest outright.
  selector:
    matchLabels:
      app: api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # one extra pod during deploy
      maxUnavailable: 0  # never go below replica count
  template:
    metadata:
      labels:
        app: api
    spec:
      terminationGracePeriodSeconds: 45
      containers:
      - name: api
        image: myapp:v2
        readinessProbe:
          httpGet:
            path: /healthz
            port: 3000
          initialDelaySeconds: 10
          periodSeconds: 5
        lifecycle:
          preStop:
            # Give the endpoints controller time to remove this pod from
            # the Service before the process starts shutting down.
            exec:
              command: ["sleep", "10"]

Kubernetes won’t route traffic to new pods until they pass readiness checks. Old pods keep serving until new ones are ready.

The preStop sleep looks silly and is the single highest-value line in that manifest. Pod termination and endpoint removal happen in parallel, not in sequence — without the pause, the kubelet can start shutting your process down while the Service is still sending it requests.

Health checks save your bacon — if they’re the right shape

#

The difference between “zero-downtime” and “zero-working-deployments” is health checks. But the useful distinction isn’t shallow versus thorough, it’s which check answers which question.

A readiness probe answers exactly one question: can this instance serve a request right now? Nothing else belongs in it.

@app.route('/healthz')
def healthz():
    # No external calls. This says "the process is up and the HTTP
    # stack works", which is the only thing readiness should mean.
    return 'OK', 200

The thorough check is still worth having — it just isn’t a probe. Put it on its own endpoint and point your monitoring at it:

@app.route('/status')
def status():
    checks = {
        'database': check_db_connection(),
        'redis': check_redis_connection(),
        'disk_space': check_disk_space(),
    }
    return jsonify(checks), 200 if all(checks.values()) else 503

Degrade, don’t disappear. If Redis is down and Redis is a cache, serve the request slowly. If the database is down and you genuinely cannot serve anything, return 503 from the request handler — the load balancer will see the errors, your alerts will fire, and the instance stays in rotation to recover on its own.

Database migrations without tears

#

Here’s where most zero-downtime efforts die. You can’t blue-green a database. You need the expand-contract pattern.

Never do this:

1. Deploy code that expects new schema
2. Run migration
3. Hope for the best

Do this instead:

1. Expand: Add new columns/tables without removing old ones
2. Deploy code that writes both and reads new-with-fallback
3. Backfill existing rows in the background
4. Deploy code that only touches the new schema
5. Contract: Remove old columns/tables

Example: renaming a column from username to email.

Migration 1 (expand) — add the column only. Nothing else:

ALTER TABLE users ADD COLUMN email VARCHAR(255);

Deploy 1 — the application writes both columns and reads the new one with a fallback:

def get_user_identifier(user):
    # New column takes precedence, old one is the fallback
    return user.email or user.username

def save_user(user_data):
    # Write to both during transition
    user.username = user_data['identifier']
    user.email = user_data['identifier']
    user.save()

Backfill — separately, in batches, outside the migration:

-- Run in a loop until zero rows are affected. Do NOT put this in the
-- migration: on a large table a single UPDATE holds locks, bloats the
-- table and blocks the deploy behind it.
UPDATE users SET email = username
WHERE email IS NULL AND id IN (
  SELECT id FROM users WHERE email IS NULL LIMIT 5000
);

Deploy 2 — the application stops touching username entirely:

def get_user_identifier(user):
    return user.email

def save_user(user_data):
    user.email = user_data['identifier']
    user.save()

Migration 2 (contract) — only now is dropping the column safe:

ALTER TABLE users DROP COLUMN username;

Three deploys for a column rename feels like a lot. It is the price of not having a maintenance window, and it’s the same price whether you’re two engineers or two hundred.

Load balancer configuration

#

Your load balancer needs to know when to stop sending traffic to instances.

For AWS ALB/ELB, use connection draining:

• Set deregistration delay to match your longest request (usually 30-60s)
• Health check interval: 10s
• Unhealthy threshold: 2 (catches issues in 20s)

For nginx open source, what you get is passive health checking — nginx notices failures as they happen to real requests:

upstream backend {
    server 10.0.1.10:3000 max_fails=2 fail_timeout=30s;
    server 10.0.1.20:3000 max_fails=2 fail_timeout=30s;
}

This marks a server as down after 2 failed requests and removes it from rotation for 30s. Note that it’s reactive by definition: some real user requests have to fail first.

Active health checks — nginx probing backends on a schedule before anyone notices, via the health_check directive — are a NGINX Plus feature and are not available in the open source build. If you need active checking without paying for Plus, that’s an argument for putting the workload behind a proxy that has it (Traefik, HAProxy, or your cloud load balancer).

Testing your deployment process

#

You can’t claim zero-downtime until you’ve tested under load.

Simple load test during deployment:

# Terminal 1: Generate load
while true; do 
    curl -w "%{http_code} %{time_total}s\n" https://api.example.com/healthz
    sleep 0.1
done

# Terminal 2: Deploy
./deploy.sh

# Watch terminal 1 - should see only 200s, no timeouts

For more realistic testing, use vegeta or k6:

echo "GET https://api.example.com/products" | \
    vegeta attack -duration=5m -rate=100 | \
    vegeta report

Run this during deployment. Success metrics:

• Zero 5xx errors
• 99th percentile latency stays under 2x normal
• No connection timeouts

When zero-downtime isn’t worth it

#

Some reality checks:

Skip zero-downtime deployments if:

• You deploy once a month
• Your app has natural maintenance windows (B2B with weekend downtime)
• The complexity exceeds the benefit (2-person team, 10 users)

Zero-downtime is about customer experience, not engineering pride. If your customers don’t care, spend the effort elsewhere.

Operations checklist

#

Before claiming victory:

• Readiness probe is shallow — no database, no cache, no external calls
• Dependency checks live on a separate endpoint that monitoring reads
• Pods have a preStop delay and a grace period longer than your slowest request
• Load balancer drains connections gracefully
• Schema changes are planned as three deploys, and the middle one is verified before the contract migration
• Backfills run in batches, outside migrations
• Deployment process is scripted, not manual
• You’ve tested deploys under real load
• Rollback procedure takes <5 minutes
• Monitoring alerts on failed deployments
• Team knows the runbook when things go wrong

Related reads

#

• CI/CD for Small Teams - Setting up the pipeline
• Database Backup Strategies - Because rollbacks need data too
• Incident Response for Small Teams - When zero-downtime becomes some-downtime