Your Deployment Platform Got Breached — Now What? A Secret Rotation Playbook

So you just got an email from your hosting provider saying there was a security incident. Environment variables — the ones holding your API keys, database credentials, and third-party tokens — might have been exposed. Your stomach drops.

I've been through this twice now. Once with a self-hosted CI runner that got compromised, and more recently when a major deployment platform disclosed that a threat actor accessed internal systems. The panic is real, but the response doesn't have to be chaotic.

Here's the step-by-step playbook I follow every time.

Step 1: Don't Panic, But Don't Wait Either

The instinct is to either freeze or start rotating every credential you've ever created. Neither is productive. What you need is a triage list.

Start by answering these questions:

• What secrets were stored in the affected platform's environment variables?
• Which of those secrets grant access to sensitive data (databases, payment processors, user PII)?
• Which secrets are scoped or limited (read-only API keys, public-facing tokens)?

I keep a simple inventory for exactly this scenario:

# secrets-inventory.sh — run this to dump a list of env var NAMES (not values)
# from your deployment config for triage purposes

#!/bin/bash

echo "=== Secrets Inventory ==="
echo "Checking .env files..."
for envfile in $(find . -name '.env*' -not -path './node_modules/*'); do
  echo "\n--- $envfile ---"
  # Print only the key names, never the values
  grep -v '^#' "$envfile" | grep '=' | cut -d'=' -f1
done

echo "\n=== Categorize each as HIGH / MEDIUM / LOW ==="
echo "HIGH: DB credentials, payment keys, auth signing secrets"
echo "MEDIUM: Third-party API keys with write access"
echo "LOW: Read-only keys, public tokens, analytics IDs"

The HIGH category gets rotated immediately. No discussion, no waiting for the incident report to finalize. Rotate first, investigate later.

Step 2: Rotate the Critical Secrets

Here's the order I follow, and it matters:

For each rotation, the process is the same:

# Example: rotating a PostgreSQL password and updating your app

# 1. Generate a new secure password
NEW_PASS=$(openssl rand -base64 32)

# 2. Update it in PostgreSQL
psql -U admin -d postgres -c "ALTER USER app_user WITH PASSWORD '$NEW_PASS';"

# 3. Update your secret store (example using a generic CLI)
# Replace with your actual secrets manager command
echo "Update DATABASE_PASSWORD in your deployment platform's env vars"
echo "New value: $NEW_PASS"

# 4. Trigger a redeployment so the app picks up the new credential
# Most platforms: push an empty commit or use their CLI
git commit --allow-empty -m "chore: rotate db credentials" && git push

# 5. Verify the app connects successfully
curl -sf https://yourapp.com/api/health | jq '.database'

The key mistake I see people make: they rotate the secret in one place but forget to update it everywhere it's referenced. If your staging environment uses the same database password as production (please don't, but I've seen it), you need to rotate both.

Step 3: Check for Unauthorized Usage

Rotating secrets stops future abuse, but you need to check whether the exposed credentials were already used. This is where logs become your best friend.

# Check for unusual database connections in the last 7 days
# PostgreSQL example — look for unfamiliar IPs or connection spikes

psql -U admin -d postgres -c "
SELECT client_addr, 
       usename, 
       count(*) as connection_count,
       min(backend_start) as first_seen,
       max(backend_start) as last_seen
FROM pg_stat_activity 
WHERE backend_start > now() - interval '7 days'
GROUP BY client_addr, usename 
ORDER BY connection_count DESC;
"

# For API keys, check your provider's dashboard for:
# - Requests from unfamiliar IPs
# - Unusual request volumes
# - Access to endpoints your app doesn't normally hit

Most third-party services (Stripe, AWS, etc.) provide audit logs. Go check them. If you see requests you don't recognize, that's your escalation trigger — you may need to notify users or report a data breach depending on your jurisdiction.

Step 4: Harden Your Secret Management Going Forward

Every breach is a forcing function to improve. Here's what I've implemented after going through this:

Stop storing long-lived secrets in env vars

Environment variables in deployment platforms are convenient, but they're a single point of failure. Move critical secrets to a dedicated secrets manager.

• HashiCorp Vault — the gold standard for self-hosted secret management
• SOPS (Secrets OPerationS) — encrypts secret files with AWS KMS, GCP KMS, or PGP
• Doppler or similar managed secret stores that provide rotation and audit trails

Use short-lived credentials where possible

Instead of a static database password that lives forever, use IAM-based authentication or short-lived tokens:

• AWS RDS supports IAM authentication — tokens expire every 15 minutes
• Google Cloud SQL supports IAM database authentication
• For internal services, mutual TLS with auto-rotating certificates beats shared secrets

Implement least privilege religiously

That API key your app uses — does it really need full admin access? Probably not.

• Scope API keys to only the permissions your app actually uses
• Use read-only credentials for services that only need to read
• Create separate credentials for production, staging, and development

Set up secret scanning in your CI pipeline

# Example GitHub Actions workflow for secret scanning
# Uses the open-source tool gitleaks

name: Secret Scan
on: [push, pull_request]

jobs:
  gitleaks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for thorough scanning
      - uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

This won't prevent a platform-level breach, but it catches the embarrassingly common case of accidentally committing secrets to your repo.

Step 5: Build a Rotation Runbook Before You Need One

The worst time to figure out your rotation process is during an active incident. Build the runbook now:

• Document every secret your app uses, where it's stored, and how to rotate it
• Automate rotation for secrets that support it (most cloud providers offer this)
• Test the rotation process — actually rotate a credential in staging and verify your app recovers
• Set calendar reminders to rotate secrets on a schedule, even without an incident

I keep a simple markdown file in our internal wiki that lists every secret, its rotation procedure, and who's responsible. It's not glamorous, but when the next breach notification lands in your inbox at 11 PM, you'll be glad it exists.

The Uncomfortable Truth

You can do everything right and still get caught in someone else's breach. Your deployment platform, your CI provider, your DNS host — any of these can be compromised. The question isn't whether it'll happen, but whether you'll be ready when it does.

The good news: if your secrets are short-lived, scoped to minimum permissions, and you have a practiced rotation process, a breach notification goes from a five-alarm fire to a Tuesday afternoon task.

Start with the inventory script above. You might be surprised how many long-lived, over-privileged secrets are sitting in your environment variables right now.