Vault Integration

Vault PKI Integration

Table of Contents

• Overview
• Service Vault Integration
• SSL/TLS Certificate Management
• Vault Commands
• Vault Auto-Unseal
  • How It Works
  • Initial Setup
  • Auto-Unseal Process
  • Manual Operations

---

Overview

HashiCorp Vault provides centralized secrets management and Public Key Infrastructure (PKI) for services. Instead of storing passwords in .env files, services fetch credentials from Vault at startup.

Benefits:

• ✅ Centralized secrets management
• ✅ Dynamic certificate generation
• ✅ Automatic certificate rotation
• ✅ Audit trail of secret access
• ✅ Optional SSL/TLS for encrypted connections
• ✅ No plaintext passwords in configuration files

Architecture:

Vault PKI Hierarchy
├── Root CA (10-year validity)
│   └── Intermediate CA (5-year validity)
│       └── Service Certificates (1-year validity)
│           ├── PostgreSQL
│           ├── MySQL
│           ├── Redis
│           └── Other services

Service Vault Integration

ALL services use Vault integration with AppRole authentication for credentials management. AppRole migration completed November 2025 (100% of services).

Integrated Services (All using AppRole):

• ✅ PostgreSQL (configs/postgres/scripts/init-approle.sh) - AppRole authentication
• ✅ MySQL (configs/mysql/scripts/init-approle.sh) - AppRole authentication
• ✅ Redis Cluster (3 nodes) (configs/redis/scripts/init-approle.sh) - AppRole authentication
• ✅ RabbitMQ (configs/rabbitmq/scripts/init-approle.sh) - AppRole authentication
• ✅ MongoDB (configs/mongodb/scripts/init-approle.sh) - AppRole authentication
• ✅ Forgejo (configs/forgejo/scripts/init-approle.sh) - AppRole authentication
• ✅ Reference API - FastAPI (reference-apps/fastapi/app/services/vault.py) - AppRole authentication

Security Improvement: Zero root token usage in containers. All services use role-based access control with least-privilege principle.

How It Works (using PostgreSQL as example with AppRole):

1. Container Startup → AppRole wrapper script (/init/init-approle.sh) executes
2. Wait for Vault → Script waits for Vault to be unsealed and ready
3. AppRole Authentication → Login to Vault using role-id and secret-id from /vault-approles/postgres/
4. Obtain Service Token → Receive short-lived service token (hvs.CAESIE... prefix)
5. Fetch Credentials & TLS Setting → GET /v1/secret/data/postgres using service token (includes tls_enabled field)
6. Validate Certificates → Check pre-generated certificates exist if TLS enabled
7. Configure PostgreSQL → Injects credentials and TLS configuration
8. Start Service → PostgreSQL starts with Vault-managed credentials

Wrapper Script (configs/postgres/scripts/init-approle.sh):

#!/bin/bash
# PostgreSQL initialization with Vault integration

# 1. Wait for Vault to be ready
wait_for_vault()

# 2. Fetch credentials AND tls_enabled from Vault
export POSTGRES_USER=$(vault_api | jq -r '.data.data.user')
export POSTGRES_PASSWORD=$(vault_api | jq -r '.data.data.password')
export POSTGRES_DB=$(vault_api | jq -r '.data.data.database')
export ENABLE_TLS=$(vault_api | jq -r '.data.data.tls_enabled // "false"')

# 3. If TLS enabled, validate pre-generated certificates
if [ "$ENABLE_TLS" = "true" ]; then
    validate_certificates  # Check certs exist in mounted volume
    configure_tls          # Configure PostgreSQL SSL
fi

# 4. Start PostgreSQL with injected credentials
exec docker-entrypoint.sh postgres

Fetching PostgreSQL Password:

# Via management script
./devstack vault-show-password postgres

# Via Vault CLI
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=$(cat ~/.config/vault/root-token)
vault kv get -field=password secret/postgres

# Via curl
curl -H "X-Vault-Token: $VAULT_TOKEN" \
  http://localhost:8200/v1/secret/data/postgres \
  | jq -r '.data.data.password'

Environment Variables:

# In .env file
VAULT_ADDR=http://vault:8200
VAULT_TOKEN=hvs.xxxxxxxxxxxxx  # From ~/.config/vault/root-token
# NOTE: TLS settings are now in Vault, not .env

Note: ALL service passwords and TLS settings have been removed from .env. All credentials and TLS configuration are now managed entirely by Vault.

AppRole Authentication (Reference API Example)

Overview: The reference-api demonstrates AppRole authentication, HashiCorp's recommended method for applications to authenticate to Vault. This is more secure than using root tokens as it provides:

• Limited, application-specific permissions
• Renewable and revocable tokens
• Audit trail of application authentication
• No root token exposure in environment variables

How It Works:

1. AppRole Credentials Mounted → Credentials stored at ~/.config/vault/approles/reference-api/

   • role-id: Identifies the AppRole (similar to username)
   • secret-id: Secret credential (similar to password)

2. Container Startup → Docker mounts credentials into container

   volumes:
     - ${HOME}/.config/vault/approles/reference-api:/vault-approles/reference-api:ro

3. Application Initialization → VaultClient reads credentials and authenticates

   # reference-apps/fastapi/app/services/vault.py
   def _login_with_approle(self) -> str:
       # Read role-id and secret-id from filesystem
       role_id_path = os.path.join(settings.VAULT_APPROLE_DIR, "role-id")
       secret_id_path = os.path.join(settings.VAULT_APPROLE_DIR, "secret-id")
   
       with open(role_id_path, 'r') as f:
           role_id = f.read().strip()
       with open(secret_id_path, 'r') as f:
           secret_id = f.read().strip()
   
       # Exchange credentials for Vault token
       url = urljoin(f"{self.vault_addr}/", "v1/auth/approle/login")
       payload = {"role_id": role_id, "secret_id": secret_id}
   
       response = client.post(url, json=payload, timeout=5.0)
       data = response.json()
       client_token = data.get("auth", {}).get("client_token")
   
       return client_token  # Token has "hvs." prefix (not root token)

4. Fallback Mechanism → If AppRole authentication fails, falls back to token-based auth

   def __init__(self):
       if settings.VAULT_APPROLE_DIR and os.path.exists(settings.VAULT_APPROLE_DIR):
           try:
               self.vault_token = self._login_with_approle()
               logger.info("Successfully authenticated to Vault using AppRole")
           except Exception as e:
               logger.warning(f"AppRole authentication failed: {e}, falling back to token-based auth")
               self.vault_token = settings.VAULT_TOKEN
       else:
           logger.info("Using token-based authentication (AppRole directory not found)")
           self.vault_token = settings.VAULT_TOKEN

Verifying AppRole Authentication:

# 1. Check AppRole credentials exist
ls -la ~/.config/vault/approles/reference-api/
# Should show: role-id, secret-id

# 2. Verify container has no VAULT_TOKEN
docker exec dev-reference-api env | grep VAULT_TOKEN
# Should be empty (no output)

# 3. Check container logs for authentication
docker logs dev-reference-api 2>&1 | grep -i "approle"
# Should show: "Successfully authenticated to Vault using AppRole"

# 4. Verify token type
docker exec dev-reference-api python -c "from app.services.vault import vault_client; print(vault_client.vault_token[:4])"
# Should show: "hvs." (AppRole token, not root token starting with "hvs.")

AppRole Setup (for reference):

AppRole credentials are created during ./devstack vault-bootstrap:

# Create AppRole
vault auth enable approle
vault write auth/approle/role/reference-api \
    token_policies="reference-api-policy" \
    token_ttl=1h \
    token_max_ttl=4h

# Generate credentials
vault read -field=role_id auth/approle/role/reference-api/role-id > ~/.config/vault/approles/reference-api/role-id
vault write -f -field=secret_id auth/approle/role/reference-api/secret-id > ~/.config/vault/approles/reference-api/secret-id

Benefits:

• Security: Application never has root token
• Audit: All API calls logged with AppRole identity
• Rotation: secret-id can be rotated without code changes
• Least Privilege: Policy limits what secrets application can access
• Production-Ready: Recommended by HashiCorp for application authentication

See Also:

• Reference API documentation: reference-apps/fastapi/README.md
• AppRole implementation: reference-apps/fastapi/app/services/vault.py
• Docker configuration: docker-compose.yml (line 879)

SSL/TLS Certificate Management

TLS Implementation: Pre-Generated Certificates with Vault-Based Configuration

The system uses a modern, production-ready TLS architecture where:

• ✅ TLS settings are stored in Vault (not environment variables)
• ✅ Certificates are pre-generated and validated before service startup
• ✅ Runtime enable/disable without container rebuilds
• ✅ All 8 services support TLS (PostgreSQL, MySQL, Redis cluster, RabbitMQ, MongoDB, FastAPI reference app)
• ✅ Dual-mode operation (accepts both SSL and non-SSL connections)

One-Time Certificate Generation:

# 1. Ensure Vault is running and bootstrapped
docker compose up -d vault
sleep 10

# 2. Bootstrap Vault (creates secrets with tls_enabled field)
VAULT_ADDR=http://localhost:8200 \
VAULT_TOKEN=$(cat ~/.config/vault/root-token) \
  bash configs/vault/scripts/vault-bootstrap.sh

# 3. Generate all certificates (stored in ~/.config/vault/certs/)
VAULT_ADDR=http://localhost:8200 \
VAULT_TOKEN=$(cat ~/.config/vault/root-token) \
  bash scripts/generate-certificates.sh

Enabling TLS for a Service (Runtime Configuration):

# 1. Set tls_enabled=true in Vault
TOKEN=$(cat ~/.config/vault/root-token)
curl -sf -X POST \
  -H "X-Vault-Token: $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"data":{"tls_enabled":true}}' \
  http://localhost:8200/v1/secret/data/postgres

# 2. Restart the service (picks up new setting)
docker restart dev-postgres

# 3. Verify TLS is enabled
docker logs dev-postgres | grep "tls_enabled"
# Should show: tls_enabled=true

Disabling TLS:

# Set tls_enabled=false and restart
TOKEN=$(cat ~/.config/vault/root-token)
curl -sf -X POST \
  -H "X-Vault-Token: $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"data":{"tls_enabled":false}}' \
  http://localhost:8200/v1/secret/data/postgres

docker restart dev-postgres

Certificate Rotation:

# 1. Delete old certificates for a service
rm -rf ~/.config/vault/certs/postgres/

# 2. Regenerate certificates
VAULT_ADDR=http://localhost:8200 \
VAULT_TOKEN=$(cat ~/.config/vault/root-token) \
  bash scripts/generate-certificates.sh

# 3. Restart service to pick up new certificates
docker restart dev-postgres

Certificate Details:

• Validity: 1 year (8760 hours)
• Storage: ~/.config/vault/certs/{service}/
• Mount: Read-only bind mounts into containers
• Format: Service-specific (e.g., MySQL uses .pem, MongoDB uses combined cert+key)

Testing TLS Connections:

All services are configured for dual-mode TLS (accepting both encrypted and unencrypted connections).

PostgreSQL:

# Get password from Vault
export PGPASSWORD=$(python3 scripts/read-vault-secret.py postgres password)

# SSL connection (with certificate verification)
psql "postgresql://dev_admin@localhost:5432/dev_database?sslmode=require"

# Non-SSL connection (dual-mode allows this)
psql "postgresql://dev_admin@localhost:5432/dev_database?sslmode=disable"

# Verify SSL is enabled
docker exec dev-postgres psql -U dev_admin -d dev_database -c "SHOW ssl;"

MySQL:

# Get password from Vault
MYSQL_PASS=$(export VAULT_TOKEN=$(cat ~/.config/vault/root-token); python3 scripts/read-vault-secret.py mysql password)

# SSL connection
mysql -h localhost -u dev_admin -p$MYSQL_PASS --ssl-mode=REQUIRED dev_database

# Non-SSL connection
mysql -h localhost -u dev_admin -p$MYSQL_PASS --ssl-mode=DISABLED dev_database

# Verify TLS is configured
docker logs dev-mysql | grep "Channel mysql_main configured to support TLS"

Redis:

# Get password from Vault
REDIS_PASS=$(export VAULT_TOKEN=$(cat ~/.config/vault/root-token); python3 scripts/read-vault-secret.py redis-1 password)

# SSL connection (TLS port 6380)
redis-cli -h localhost -p 6380 --tls \
  --cacert ~/.config/vault/certs/redis-1/ca.crt \
  --cert ~/.config/vault/certs/redis-1/redis.crt \
  --key ~/.config/vault/certs/redis-1/redis.key \
  -a $REDIS_PASS PING

# Non-SSL connection (standard port 6379)
redis-cli -h localhost -p 6379 -a $REDIS_PASS PING

# Verify dual ports
docker logs dev-redis-1 | grep "Ready to accept connections"

RabbitMQ:

# SSL port: 5671
# Non-SSL port: 5672 (management UI also available on 15672)

# Test management API
curl -u admin:$(export VAULT_TOKEN=$(cat ~/.config/vault/root-token); python3 scripts/read-vault-secret.py rabbitmq password) \
  http://localhost:15672/api/overview

MongoDB:

# Get credentials from Vault
MONGO_USER=$(export VAULT_TOKEN=$(cat ~/.config/vault/root-token); python3 scripts/read-vault-secret.py mongodb user)
MONGO_PASS=$(export VAULT_TOKEN=$(cat ~/.config/vault/root-token); python3 scripts/read-vault-secret.py mongodb password)

# SSL connection (if TLS is enabled)
mongosh "mongodb://$MONGO_USER:$MONGO_PASS@localhost:27017/dev_database?tls=true&tlsCAFile=$HOME/.config/vault/certs/mongodb/ca.pem"

# Non-SSL connection
mongosh "mongodb://$MONGO_USER:$MONGO_PASS@localhost:27017/dev_database"

SSL/TLS Modes:

• PostgreSQL SSL Modes:

  • disable - No SSL
  • allow - Try SSL, fallback to plain
  • prefer - Prefer SSL, fallback to plain
  • require - Require SSL (no cert verification)
  • verify-ca - Require SSL + verify CA certificate
  • verify-full - Require SSL + verify CA + hostname matching

• MySQL SSL Modes:

  • DISABLED - No SSL
  • PREFERRED - Use SSL if available
  • REQUIRED - Require SSL
  • VERIFY_CA - Verify CA certificate
  • VERIFY_IDENTITY - Verify CA + hostname

Vault Commands

Vault Management Script Commands:

# Initialize Vault (first time only)
./devstack vault-init

# Check Vault status
./devstack vault-status

# Get root token
./devstack vault-token

# Unseal Vault manually (if needed)
./devstack vault-unseal

# Bootstrap PKI and service credentials
./devstack vault-bootstrap

# Export CA certificates
./devstack vault-ca-cert

# Show service password
./devstack vault-show-password postgres
./devstack vault-show-password mysql

Vault Bootstrap Process:

The vault-bootstrap command sets up the complete PKI infrastructure:

1. Generate Root CA (if not exists)
2. Generate Intermediate CA CSR
3. Sign Intermediate CA with Root CA
4. Install Intermediate CA certificate
5. Create PKI roles for each service (postgres-role, mysql-role, etc.)
6. Generate and store service credentials (user, password, database)
7. Export CA certificates to ~/.config/vault/ca/

Manual Vault Operations:

# Set environment
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=$(cat ~/.config/vault/root-token)

# List secret paths
vault kv list secret/

# Get PostgreSQL credentials
vault kv get secret/postgres

# Update password (manual rotation)
vault kv put secret/postgres \
  user=dev_admin \
  password=new_generated_password \
  database=dev_database

# Issue certificate manually
vault write pki_int/issue/postgres-role \
  common_name=postgres.dev-services.local \
  ttl=8760h

# View PKI role configuration
vault read pki_int/roles/postgres-role

PKI Certificate Paths:

Vault PKI Endpoints:
├── /v1/pki/ca/pem                      # Root CA certificate
├── /v1/pki_int/ca/pem                  # Intermediate CA certificate
├── /v1/pki_int/roles/postgres-role     # PostgreSQL certificate role
├── /v1/pki_int/issue/postgres-role     # Issue PostgreSQL certificate
└── /v1/secret/data/postgres            # PostgreSQL credentials

Credential Loading for Non-Container Services:

For services that need Vault credentials but aren't containerized (e.g., PgBouncer, Forgejo), credentials are loaded via environment variables:

Script: scripts/load-vault-env.sh

This script loads credentials from Vault into environment variables for docker-compose:

#!/bin/bash
# Load credentials from Vault and export as environment variables

# 1. Wait for Vault to be ready
# 2. Read VAULT_TOKEN from ~/.config/vault/root-token
# 3. Fetch PostgreSQL password: secret/postgres
# 4. Export POSTGRES_PASSWORD for docker-compose

export POSTGRES_PASSWORD=$(python3 scripts/read-vault-secret.py postgres password)

Script: scripts/read-vault-secret.py

Python helper to read secrets from Vault KV v2 API:

#!/usr/bin/env python3
# Usage: read-vault-secret.py <path> <field>
# Example: read-vault-secret.py postgres password

import sys, json, urllib.request, os

vault_addr = os.getenv('VAULT_ADDR', 'http://localhost:8200')
vault_token = os.getenv('VAULT_TOKEN')

url = f"{vault_addr}/v1/secret/data/{sys.argv[1]}"
req = urllib.request.Request(url)
req.add_header('X-Vault-Token', vault_token)

with urllib.request.urlopen(req) as response:
    data = json.loads(response.read().decode())
    print(data['data']['data'][sys.argv[2]])

When Credentials Are Loaded:

The devstack.py script automatically loads credentials during startup:

1. Start Vault container
2. Wait 5 seconds for Vault to be ready
3. Run source scripts/load-vault-env.sh
4. Export credentials as environment variables
5. Start remaining services with injected credentials

All Services Are Now Vault-Integrated:

✅ All database services (PostgreSQL, MySQL, MongoDB) ✅ All caching services (Redis Cluster) ✅ All message queue services (RabbitMQ) ✅ All connection pooling services (PgBouncer)

No migration needed - the infrastructure is complete!

Vault Auto-Unseal

How It Works

Vault runs in two concurrent processes within the container:

Container: dev-vault
├── Process 1: vault server
│   - Listens on 0.0.0.0:8200
│   - Uses file storage: /vault/data
│   - Config: /vault/config/vault.hcl
│
└── Process 2: vault-auto-unseal.sh
    - Waits for Vault to be ready
    - Unseals using saved keys
    - Sleeps indefinitely (no CPU overhead)

Entrypoint (docker-compose.yml:360-366):

entrypoint: >
  sh -c "
  chown -R vault:vault /vault/data &&
  docker-entrypoint.sh server &
  /usr/local/bin/vault-auto-unseal.sh &
  wait -n
  "

Process Flow:

1. Fix /vault/data permissions (chown)
2. Start Vault server in background (&)
3. Start auto-unseal script in background (&)
4. Wait for either process to exit (wait -n)

Initial Setup

First-Time Initialization:

./configs/vault/scripts/vault-init.sh
# Or
./devstack vault-init

What Happens:

1. Waits for Vault to be ready (max 30 seconds)
2. Checks if already initialized
3. If not initialized:
   • POSTs to /v1/sys/init with {"secret_shares": 5, "secret_threshold": 3}
   • Receives 5 unseal keys + root token
   • Saves to ~/.config/vault/keys.json (chmod 600)
   • Saves root token to ~/.config/vault/root-token (chmod 600)
4. Unseals Vault using 3 of 5 keys
5. Displays status and root token

Shamir Secret Sharing:

• 5 keys generated
• Any 3 keys can unseal Vault
• Designed for distributed trust (give keys to different people/systems)
• Lost keys = cannot unseal (data is encrypted and unrecoverable)

Auto-Unseal Process

Script (configs/vault/scripts/vault-auto-unseal.sh):

# 1. Wait for Vault API (max 30 attempts, 1s each)
wget --spider http://127.0.0.1:8200/v1/sys/health?uninitcode=200&sealedcode=200

# 2. Check seal status
wget -qO- http://127.0.0.1:8200/v1/sys/seal-status
# → {"sealed": true}

# 3. Read unseal keys from mounted volume
cat /vault-keys/keys.json | extract 3 keys

# 4. POST each key to unseal endpoint
for key in key1 key2 key3; do
  wget --post-data='{"key":"'$key'"}' http://127.0.0.1:8200/v1/sys/unseal
done

# 5. Verify unsealed
wget -qO- http://127.0.0.1:8200/v1/sys/seal-status
# → {"sealed": false}

# 6. Sleep indefinitely (no monitoring overhead)
while true; do sleep 3600; done

Why Not Continuous Monitoring?

• Original design had 10-second checks (360 API calls/hour)
• Optimized to single unseal + sleep
• Saves 99% of API calls and CPU cycles
• Trade-off: Won't auto-reseal if manually sealed (must restart container)

Manual Operations

Check Vault Status:

./devstack vault-status

# Or directly
export VAULT_ADDR=http://localhost:8200
export VAULT_TOKEN=$(cat ~/.config/vault/root-token)
vault status

Manually Unseal:

./devstack vault-unseal

# Or using vault CLI
vault operator unseal  # Repeat 3 times with different keys

Seal Vault:

vault operator seal
# Note: Won't auto-reseal until container restarts

Rotate Root Token:

vault token create -policy=root
# Save new token to ~/.config/vault/root-token

Backup Unseal Keys:

# Encrypt and backup
tar czf vault-keys-$(date +%Y%m%d).tar.gz ~/.config/vault/
gpg -c vault-keys-*.tar.gz
# Store encrypted file in secure location (1Password, etc.)

---

Certificate Lifecycle Management

Certificate Expiration Timeline

Certificate Type	Validity Period	Typical Issue Date	Expiration Date	Renewal Window
Root CA (pki)	10 years	2025-01-15	2035-01-15	9 years notice
Intermediate CA (pki_int)	5 years	2025-01-15	2030-01-15	4.5 years notice
Service Certificates	1 year	2025-01-15	2026-01-15	30 days notice

Critical: Service certificates must be renewed annually. Set calendar reminders for 30 days before expiration.

Checking Certificate Expiration

Check all service certificates:

# Quick check all services
for service in postgres mysql redis-1 redis-2 redis-3 rabbitmq mongodb; do
  echo "=== $service ==="
  openssl x509 -in ~/.config/vault/certs/$service/cert.pem -noout -enddate
  echo ""
done

# With days until expiry
for service in postgres mysql redis-1 redis-2 redis-3 rabbitmq mongodb; do
  EXPIRY=$(openssl x509 -in ~/.config/vault/certs/$service/cert.pem -noout -enddate | cut -d= -f2)
  EXPIRY_EPOCH=$(date -j -f "%b %d %T %Y %Z" "$EXPIRY" +%s 2>/dev/null || date -d "$EXPIRY" +%s)
  NOW_EPOCH=$(date +%s)
  DAYS_UNTIL=$(( ($EXPIRY_EPOCH - $NOW_EPOCH) / 86400 ))
  echo "$service: $DAYS_UNTIL days until expiry"
done

Check Root CA:

openssl x509 -in ~/.config/vault/ca/ca.pem -noout -enddate -subject

Check Intermediate CA:

vault read pki_int/ca/pem | openssl x509 -noout -enddate -subject

Service Certificate Renewal

When to Renew: 30 days before expiration (or sooner)

Automated Renewal Script: scripts/renew-certificates.sh

#!/bin/bash
# Renew all service certificates from Vault PKI
# Run annually or when certificates are approaching expiration

set -e

VAULT_ADDR="${VAULT_ADDR:-http://localhost:8200}"
VAULT_TOKEN="${VAULT_TOKEN:-$(cat ~/.config/vault/root-token)}"

echo "🔄 Renewing service certificates..."
echo "Vault: $VAULT_ADDR"

# Check Vault is unsealed
if ! vault status > /dev/null 2>&1; then
  echo "❌ Error: Vault is not accessible or is sealed"
  echo "Run: ./devstack vault-unseal"
  exit 1
fi

# Backup existing certificates
echo "📦 Backing up existing certificates..."
BACKUP_DIR=~/.config/vault/certs-backup-$(date +%Y%m%d-%H%M%S)
mkdir -p "$BACKUP_DIR"
cp -r ~/.config/vault/certs/* "$BACKUP_DIR/"
echo "   Backup created: $BACKUP_DIR"

# Generate new certificates
echo "🔐 Generating new certificates..."
export VAULT_ADDR VAULT_TOKEN
./scripts/generate-certificates.sh

# Verify new certificates
echo "✅ Verifying new certificates..."
for service in postgres mysql redis-1 redis-2 redis-3 rabbitmq mongodb; do
  if [ -f ~/.config/vault/certs/$service/cert.pem ]; then
    EXPIRY=$(openssl x509 -in ~/.config/vault/certs/$service/cert.pem -noout -enddate | cut -d= -f2)
    echo "   $service: Valid until $EXPIRY"
  fi
done

echo ""
echo "🔄 Restarting services to load new certificates..."
./devstack restart

echo ""
echo "✅ Certificate renewal complete!"
echo "   Old certificates backed up to: $BACKUP_DIR"
echo "   New certificates expire in ~365 days"
echo ""
echo "📅 Set reminder to renew again in 11 months"

Make executable and run:

chmod +x scripts/renew-certificates.sh
./scripts/renew-certificates.sh

Manual Renewal (if script fails):

# 1. Set environment
export VAULT_TOKEN=$(cat ~/.config/vault/root-token)
export VAULT_ADDR=http://localhost:8200

# 2. Backup existing certificates
cp -r ~/.config/vault/certs ~/.config/vault/certs-backup-$(date +%Y%m%d)

# 3. Regenerate certificates
./scripts/generate-certificates.sh

# 4. Restart services
./devstack restart

# 5. Verify new certificates
for service in postgres mysql redis-1; do
  openssl x509 -in ~/.config/vault/certs/$service/cert.pem -noout -dates
done

Intermediate CA Renewal

When to Renew: 60 days before expiration (5-year cert, so plan at 4 years 10 months)

⚠️ CRITICAL: Intermediate CA renewal affects ALL service certificates. Plan carefully.

Renewal Procedure:

# 1. Set environment
export VAULT_TOKEN=$(cat ~/.config/vault/root-token)
export VAULT_ADDR=http://localhost:8200

# 2. Check current intermediate CA expiration
vault read pki_int/ca/pem | openssl x509 -noout -enddate

# 3. Generate new intermediate CSR
vault write -format=json pki_int/intermediate/generate/internal \
  common_name="DevStack Core Intermediate CA v2" \
  ttl="43800h" \
  key_type="rsa" \
  key_bits="4096" > pki_int_csr_v2.json

CSR=$(jq -r '.data.csr' < pki_int_csr_v2.json)

# 4. Sign with Root CA
vault write -format=json pki/root/sign-intermediate \
  csr="$CSR" \
  format=pem_bundle \
  ttl="43800h" > pki_int_cert_v2.json

CERT=$(jq -r '.data.certificate' < pki_int_cert_v2.json)

# 5. Import signed certificate
vault write pki_int/intermediate/set-signed certificate="$CERT"

# 6. Verify new intermediate CA
vault read pki_int/ca/pem | openssl x509 -noout -text | grep "Not After"

# 7. Regenerate ALL service certificates
./scripts/generate-certificates.sh

# 8. Restart all services
./devstack restart

# 9. Verify everything works
./devstack health

Post-Renewal Verification:

# Check Intermediate CA is active
vault list pki_int/roles

# Test certificate issuance
vault write pki_int/issue/postgres-role \
  common_name=test.postgres.local \
  ttl=1h

# Verify service health
./devstack health

Root CA Renewal

When to Renew: 6-12 months before expiration (10-year cert, plan at 9 years)

⚠️ MAJOR EVENT: Root CA renewal requires extensive planning:

1. Coordinated rollover period (dual root CA trust)
2. New intermediate CA must be issued
3. All service certificates must be regenerated
4. Distribution of new root CA to all clients
5. Testing in non-production environment first

DO NOT attempt Root CA renewal without:

• Full backup of current PKI
• Test environment validation
• Communication plan for dependent systems
• Rollback procedure

Recommendation: Create a dedicated ROOT_CA_RENEWAL.md runbook 6 months before expiration with detailed step-by-step procedures specific to your environment.

Emergency Root CA Renewal (if Root CA is compromised):

See DISASTER_RECOVERY.md - "Vault Data Loss" section.

Automated Expiration Monitoring

Daily Cron Job:

# Add to crontab: crontab -e
# Check certificate expiration daily at 9 AM
0 9 * * * /Users/gator/devstack-core/scripts/check-cert-expiry.sh 2>&1 | mail -s "Certificate Expiry Report" admin@example.com

Monitoring Script: scripts/check-cert-expiry.sh

#!/bin/bash
# Check certificate expiration and alert if approaching expiry

WARN_DAYS=30
CRIT_DAYS=7
FOUND_WARNING=0
FOUND_CRITICAL=0

echo "Certificate Expiration Report - $(date)"
echo "==========================================="
echo ""

# Check service certificates
echo "Service Certificates:"
for cert_path in ~/.config/vault/certs/*/cert.pem; do
  if [ ! -f "$cert_path" ]; then
    continue
  fi

  service=$(basename $(dirname "$cert_path"))
  expiry=$(openssl x509 -in "$cert_path" -noout -enddate | cut -d= -f2)

  # Calculate days until expiry (macOS compatible)
  if date -j -f "%b %d %T %Y %Z" "$expiry" +%s > /dev/null 2>&1; then
    expiry_epoch=$(date -j -f "%b %d %T %Y %Z" "$expiry" +%s)
  else
    expiry_epoch=$(date -d "$expiry" +%s)
  fi

  now_epoch=$(date +%s)
  days_until=$(( ($expiry_epoch - $now_epoch) / 86400 ))

  if [ $days_until -lt $CRIT_DAYS ]; then
    echo "❌ CRITICAL: $service certificate expires in $days_until days ($expiry)"
    FOUND_CRITICAL=1
  elif [ $days_until -lt $WARN_DAYS ]; then
    echo "⚠️  WARNING: $service certificate expires in $days_until days ($expiry)"
    FOUND_WARNING=1
  else
    echo "✅ OK: $service certificate valid for $days_until days"
  fi
done

echo ""
echo "Root and Intermediate CAs:"

# Check Root CA
if [ -f ~/.config/vault/ca/ca.pem ]; then
  root_expiry=$(openssl x509 -in ~/.config/vault/ca/ca.pem -noout -enddate | cut -d= -f2)
  echo "   Root CA expires: $root_expiry"
fi

# Check Intermediate CA
if command -v vault > /dev/null 2>&1; then
  export VAULT_ADDR=http://localhost:8200
  export VAULT_TOKEN=$(cat ~/.config/vault/root-token 2>/dev/null)
  if [ -n "$VAULT_TOKEN" ]; then
    int_expiry=$(vault read pki_int/ca/pem 2>/dev/null | openssl x509 -noout -enddate 2>/dev/null)
    if [ -n "$int_expiry" ]; then
      echo "   Intermediate CA expires: $int_expiry"
    fi
  fi
fi

echo ""
echo "==========================================="

# Exit codes for monitoring systems
if [ $FOUND_CRITICAL -eq 1 ]; then
  exit 2
elif [ $FOUND_WARNING -eq 1 ]; then
  exit 1
else
  exit 0
fi

Make executable:

chmod +x scripts/check-cert-expiry.sh

# Test run
./scripts/check-cert-expiry.sh

Certificate Revocation

When to Revoke: Certificate compromise, key exposure, or service decommissioning

Revocation Procedure:

# 1. Get certificate serial number
openssl x509 -in ~/.config/vault/certs/postgres/cert.pem -noout -serial

# Example output: serial=3A:5F:2B:...

# 2. Revoke certificate
vault write pki_int/revoke serial_number="3A:5F:2B:..."

# 3. Generate new certificate immediately
vault write pki_int/issue/postgres-role \
  common_name="postgres" \
  ttl="8760h" \
  format=pem_bundle > postgres_new.json

# 4. Extract and install new certificate
jq -r '.data.certificate' < postgres_new.json > ~/.config/vault/certs/postgres/cert.pem
jq -r '.data.private_key' < postgres_new.json > ~/.config/vault/certs/postgres/key.pem
jq -r '.data.ca_chain[]' < postgres_new.json > ~/.config/vault/certs/postgres/ca.pem

# 5. Restart affected service
docker compose restart postgres

# 6. Update Certificate Revocation List (CRL)
vault read pki_int/crl

Best Practices

1. Calendar Reminders:

   • Root CA: Set reminder 9 years from issue date
   • Intermediate CA: Set reminder 4 years 6 months from issue date
   • Service Certificates: Set reminders every 11 months

2. Automate Renewal:

   • Service certificates: Run renew-certificates.sh monthly (automated)
   • Intermediate CA: Manual process with advance planning
   • Root CA: Treat as major project, plan 6-12 months ahead

3. Test Renewal Process:

   • Practice renewal in test environment quarterly
   • Document any issues encountered
   • Update renewal scripts based on lessons learned

4. Monitor Continuously:

   • Daily automated expiration checks (check-cert-expiry.sh)
   • Alert if certificates < 30 days from expiry
   • Escalate if certificates < 7 days from expiry

5. Backup Before Renewal:

   • Always backup ~/.config/vault/ before renewal
   • Store backups securely off-system
   • Test restoration from backup

6. Verify After Renewal:

   • Check certificate dates with openssl x509
   • Verify services restart successfully
   • Run health checks: ./devstack health
   • Test TLS connections to services

Troubleshooting Certificate Issues

Problem: Service won't start after certificate renewal

Solution:

# Check certificate permissions
ls -la ~/.config/vault/certs/postgres/

# Should be readable (644 for certs, 600 for keys)
chmod 644 ~/.config/vault/certs/postgres/cert.pem
chmod 644 ~/.config/vault/certs/postgres/ca.pem
chmod 600 ~/.config/vault/certs/postgres/key.pem

# Check certificate validity
openssl verify -CAfile ~/.config/vault/certs/postgres/ca.pem \
  ~/.config/vault/certs/postgres/cert.pem

# Check service logs
docker compose logs postgres | grep -i tls

Problem: Certificate verification failed

Solution:

# Verify certificate chain
openssl verify -verbose -CAfile ~/.config/vault/ca/ca.pem \
  -untrusted ~/.config/vault/ca/ca-chain.pem \
  ~/.config/vault/certs/postgres/cert.pem

# If chain is broken, regenerate
./scripts/generate-certificates.sh

Problem: Vault won't issue new certificates

Solution:

# Check Vault PKI role exists
vault list pki_int/roles

# If missing, re-run vault-bootstrap
./devstack vault-bootstrap

# Check intermediate CA is configured
vault read pki_int/cert/ca

Renewal Checklist

30 Days Before Expiry:

• Run check-cert-expiry.sh to confirm expiry dates
• Schedule maintenance window (service restart required)
• Notify stakeholders of planned renewal
• Backup current certificates
• Test renewal script in non-production environment

Renewal Day:

• Verify Vault is healthy and unsealed
• Backup ~/.config/vault/ directory
• Run renew-certificates.sh script
• Verify new certificate dates
• Restart services: ./devstack restart
• Verify services healthy: ./devstack health
• Test TLS connections
• Monitor service logs for TLS errors

Post-Renewal (24 hours later):

• Confirm no TLS errors in logs
• Verify applications connecting successfully
• Update renewal tracking spreadsheet
• Set reminder for next renewal (11 months)
• Document any issues encountered

---