Production Setup

Deploy nself v0.3.5 to production with 100% service reliability, enhanced SSL/HTTPS support, security hardening, monitoring, backups, and high availability configurations.

✨ Enhanced in v0.3.5: Production-Ready Reliability

  • • 100% Service Reliability: All 17 services start successfully in production
  • • Complete SSL/HTTPS: Automatic Let's Encrypt certificates with green lock compatibility
  • • Enhanced SSL Management: New nself ssl commands for certificate management
  • • Volume Mount Fixes: Resolved production deployment issues

Production Preparation

Generate Production Configuration

# Generate secure production config
nself prod

# This creates:
# - .env.prod-template (production configuration)
# - .env.prod-secrets (backup of generated secrets)

# Generated secrets include:
# - Cryptographically secure passwords
# - JWT secrets and signing keys
# - Admin secrets and API keys
# - Database connection strings

Review Production Configuration

Edit .env.prod-template with your production settings:

# Domain and SSL
DOMAIN=yourdomain.com
LETSENCRYPT_EMAIL=admin@yourdomain.com
SSL_MODE=letsencrypt

# Database (use managed database in production)
POSTGRES_HOST=your-production-db-host
POSTGRES_PORT=5432
POSTGRES_DB=yourapp_production
POSTGRES_USER=yourapp_user
POSTGRES_PASSWORD=generated_secure_password

# Security
ENVIRONMENT=production
DEBUG_MODE=false
HASURA_GRAPHQL_DEV_MODE=false
HASURA_GRAPHQL_ENABLE_CONSOLE=false

# Performance
POSTGRES_MAX_CONNECTIONS=100
REDIS_MAXMEMORY=512mb
HASURA_GRAPHQL_POOL_SIZE=10

# Monitoring
LOG_LEVEL=info
METRICS_ENABLED=true
MONITORING_ENABLED=true

Server Requirements

Minimum Production Requirements

  • CPU: 4 cores (2.4+ GHz)
  • RAM: 8GB minimum, 16GB recommended
  • Storage: 50GB SSD minimum, 200GB+ recommended
  • Network: Stable internet with static IP
  • OS: Ubuntu 20.04+ LTS, CentOS 8+, or Debian 11+

Recommended Production Specs

  • CPU: 8+ cores
  • RAM: 32GB+
  • Storage: NVMe SSD with backups
  • Network: Load balancer with multiple availability zones
  • Monitoring: Dedicated monitoring and logging

Server Setup

Initial Server Configuration

# Update system
sudo apt update && sudo apt upgrade -y

# Install Docker and Docker Compose
curl -fsSL https://get.docker.com | bash
sudo usermod -aG docker $USER

# Install nself
curl -fsSL https://raw.githubusercontent.com/acamarata/nself/main/install.sh | bash

# Configure firewall
sudo ufw allow 22/tcp   # SSH
sudo ufw allow 80/tcp   # HTTP
sudo ufw allow 443/tcp  # HTTPS
sudo ufw --force enable

# Set up swap (if not already configured)
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Security Hardening

# Disable root login
sudo sed -i 's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
sudo systemctl restart sshd

# Configure automatic security updates
sudo apt install unattended-upgrades
sudo dpkg-reconfigure unattended-upgrades

# Install fail2ban
sudo apt install fail2ban
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

# Configure log rotation
sudo logrotate -d /etc/logrotate.conf

SSL/TLS Configuration Enhanced v0.3.5

nself v0.3.5 provides complete SSL/HTTPS support with automatic certificate management and production-ready configurations.

Automatic SSL Setup (Recommended)

# Bootstrap SSL certificates on production server
nself ssl bootstrap

# Configuration in .env.prod-template:
SSL_MODE=letsencrypt
LETSENCRYPT_EMAIL=admin@yourdomain.com
DOMAIN=yourdomain.com

# Optional: Additional domains
ADDITIONAL_DOMAINS=www.yourdomain.com,api.yourdomain.com

# Certificate renewal (automatic)
SSL_AUTO_RENEW=true
SSL_RENEW_DAYS=30

# SSL Management Commands (v0.3.5)
# Check certificate status
nself ssl status

# Manually renew certificates
nself ssl renew

Custom SSL Certificates

# If using custom certificates:
SSL_MODE=custom
SSL_CERT_PATH=/path/to/certificate.crt
SSL_KEY_PATH=/path/to/private.key
SSL_CHAIN_PATH=/path/to/chain.crt  # Optional

# Place certificates in ssl/ directory
mkdir -p ssl/
cp your-cert.crt ssl/
cp your-key.key ssl/
cp your-chain.crt ssl/

# Bootstrap custom certificates
nself ssl bootstrap --custom

# Verify certificate installation
nself ssl status

Database Setup

Managed Database (Recommended)

Use a managed PostgreSQL service for production:

AWS RDS Configuration

# Update .env.prod-template:
POSTGRES_HOST=your-rds-endpoint.amazonaws.com
POSTGRES_PORT=5432
POSTGRES_DB=yourapp_production
POSTGRES_USER=yourapp_user
POSTGRES_PASSWORD=secure_password
POSTGRES_SSL_MODE=require

# Connection pooling
POSTGRES_MAX_CONNECTIONS=100
POSTGRES_POOL_SIZE=10

Google Cloud SQL Configuration

POSTGRES_HOST=your-cloud-sql-ip
POSTGRES_PORT=5432
POSTGRES_DB=yourapp_production
POSTGRES_USER=yourapp_user
POSTGRES_PASSWORD=secure_password
POSTGRES_SSL_MODE=require

# Cloud SQL Proxy (if using private IP)
POSTGRES_PROXY_ENABLED=true

Self-Hosted Database

If self-hosting PostgreSQL, use these production settings:

# PostgreSQL production configuration
POSTGRES_SHARED_BUFFERS=2GB
POSTGRES_EFFECTIVE_CACHE_SIZE=6GB
POSTGRES_WORK_MEM=64MB
POSTGRES_MAINTENANCE_WORK_MEM=512MB
POSTGRES_MAX_CONNECTIONS=200
POSTGRES_WAL_BUFFERS=16MB
POSTGRES_CHECKPOINT_COMPLETION_TARGET=0.9

# Enable connection pooling
PGBOUNCER_ENABLED=true
PGBOUNCER_POOL_SIZE=25

Deployment Process

Initial Deployment

# 1. Copy production configuration
cp .env.prod-template .env

# 2. Build production environment
nself build

# 3. Initialize database (first time only)
nself db migrate:up

# 4. Seed production data
nself db seed --env production

# 5. Start all services
nself up --detach

# 6. Verify deployment
nself doctor
nself status --verbose

Deployment Verification

# Check all services are running
nself status

# Test connectivity
curl -k https://yourdomain.com/healthz
curl -k https://yourdomain.com/v1/graphql

# Check SSL certificate
openssl s_client -connect yourdomain.com:443 -servername yourdomain.com

# Verify database connection
nself exec postgres pg_isready -U postgres

Monitoring and Observability

Enable Production Monitoring

# In .env production:
MONITORING_ENABLED=true
METRICS_ENABLED=true
LOG_LEVEL=info

# Optional monitoring services
GRAFANA_ENABLED=true
PROMETHEUS_ENABLED=true
ALERTMANAGER_ENABLED=true

# External monitoring integrations
DATADOG_API_KEY=your-datadog-key
NEWRELIC_LICENSE_KEY=your-newrelic-key

Health Checks and Alerts

# Set up health check cron job
crontab -e

# Add this line:
*/5 * * * * /usr/local/bin/nself doctor > /tmp/nself-health.log 2>&1 || echo "nself health check failed" | mail -s "Production Alert" admin@yourdomain.com

# Configure Grafana alerts
# CPU usage > 80%
# Memory usage > 85%  
# Disk usage > 90%
# Database connections > 80% of max
# API error rate > 5%

Backup Strategy

Automated Database Backups

# Daily backup script
#!/bin/bash
# /usr/local/bin/backup-database.sh

BACKUP_DIR="/backup/database"
DATE=$(date +%Y%m%d_%H%M%S)
RETENTION_DAYS=30

# Create backup directory
mkdir -p $BACKUP_DIR

# Create database backup
nself db backup --name "daily_$DATE"

# Compress and move to backup location
tar -czf "$BACKUP_DIR/backup_$DATE.tar.gz" bin/dbsyncs/

# Clean old backups
find $BACKUP_DIR -name "backup_*.tar.gz" -mtime +$RETENTION_DAYS -delete

# Log backup completion
echo "$(date): Database backup completed - backup_$DATE.tar.gz" >> /var/log/nself-backup.log

Backup Cron Jobs

# Add to crontab (crontab -e):

# Daily database backup at 2 AM
0 2 * * * /usr/local/bin/backup-database.sh

# Weekly full system backup at 3 AM Sunday
0 3 * * 0 /usr/local/bin/backup-full-system.sh

# Monthly backup verification
0 4 1 * * /usr/local/bin/verify-backups.sh

Security Configuration

Network Security

# Configure firewall rules
sudo ufw reset
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp   # SSH (consider changing port)
sudo ufw allow 80/tcp   # HTTP (redirects to HTTPS)
sudo ufw allow 443/tcp  # HTTPS
sudo ufw enable

# Optional: Change SSH port
sudo sed -i 's/#Port 22/Port 2222/' /etc/ssh/sshd_config
sudo systemctl restart sshd
sudo ufw allow 2222/tcp
sudo ufw delete allow 22/tcp

Application Security

# Security headers and CORS
CORS_ORIGINS=https://yourdomain.com,https://www.yourdomain.com
SECURITY_HEADERS_ENABLED=true
HSTS_ENABLED=true
XSS_PROTECTION_ENABLED=true

# Rate limiting
RATE_LIMITING_ENABLED=true
RATE_LIMIT_MAX_REQUESTS=100
RATE_LIMIT_WINDOW=900  # 15 minutes

# DDoS protection
DDOS_PROTECTION_ENABLED=true
DDOS_MAX_CONNECTIONS=1000

Performance Optimization

Resource Allocation

# Optimize service resource limits
POSTGRES_MEMORY_LIMIT=4GB
HASURA_MEMORY_LIMIT=2GB
REDIS_MEMORY_LIMIT=1GB
NGINX_MEMORY_LIMIT=512MB

# CPU limits
POSTGRES_CPU_LIMIT=2.0
HASURA_CPU_LIMIT=1.0
REDIS_CPU_LIMIT=0.5

Caching Configuration

# Redis cache settings
REDIS_MAXMEMORY=1gb
REDIS_MAXMEMORY_POLICY=allkeys-lru
REDIS_CACHE_TTL=3600

# HTTP caching
NGINX_CACHE_ENABLED=true
NGINX_CACHE_SIZE=1g
NGINX_CACHE_INACTIVE=24h

High Availability

Load Balancer Configuration

# Multiple server setup
SERVER_1=server1.yourdomain.com
SERVER_2=server2.yourdomain.com
LOAD_BALANCER=lb.yourdomain.com

# Health check endpoints
HEALTH_CHECK_PATH=/healthz
HEALTH_CHECK_INTERVAL=30s
HEALTH_CHECK_TIMEOUT=10s

Database Replication

# Master-slave PostgreSQL setup
POSTGRES_REPLICATION_ENABLED=true
POSTGRES_MASTER_HOST=db-master.yourdomain.com
POSTGRES_REPLICA_HOST=db-replica.yourdomain.com

# Read-only replica for queries
POSTGRES_READ_REPLICA_ENABLED=true

Maintenance and Updates

Update Strategy

# Scheduled maintenance window
# 1. Backup current state
nself backup --name "pre-update-$(date +%Y%m%d)"

# 2. Update nself CLI
nself update

# 3. Apply any new migrations
nself db migrate:up

# 4. Restart services with new configuration
nself restart

# 5. Verify everything is working
nself doctor
nself status --verbose

Zero-Downtime Updates

# For critical production systems
# 1. Set up blue-green deployment
# 2. Update standby environment
# 3. Switch traffic to updated environment  
# 4. Keep old environment as fallback

# Rolling updates for stateless services
nself update --rolling
nself restart --rolling hasura

Disaster Recovery

Recovery Procedures

# Complete system recovery
# 1. Provision new server
# 2. Install nself and dependencies  
# 3. Restore configuration files
# 4. Restore database from backup
nself db restore /path/to/backup.sql
# 5. Start services
nself up
# 6. Update DNS if needed

Recovery Testing

Regularly test your disaster recovery procedures:

  • Monthly: Test backup restoration
  • Quarterly: Full disaster recovery drill
  • Document: Recovery time objectives (RTO) and recovery point objectives (RPO)
  • Automate: As much of the recovery process as possible

Next Steps