Troubleshooting Guide

This comprehensive troubleshooting guide covers common issues you might encounter with nself and provides step-by-step solutions to resolve them quickly.

✨ Much Improved in v0.3.5: Reliability Fixes

Most common issues from previous versions have been resolved:

• Volume Mount Issues: Fixed critical Docker volume mounting problems
• Service Startup Failures: 100% service reliability (17/17 services start successfully)
• SSL Certificate Problems: Complete SSL automation with nself trust and nself ssl commands
• Port Conflicts: Smart port allocation prevents most conflicts
• Node.js Module Loading: Improved module resolution and loading

First Steps

Before diving into specific issues, always start with nself doctor to get an overview of potential problems in your setup.

General Diagnostic Commands

Note: Some advanced diagnostic commands shown below are planned for future releases. Currently available commands are marked with ✓.

# ✓ Check nself version and help
nself version
nself help

# ✓ View current configuration (.env.local)
cat .env.local

# Planned for future release:
# nself doctor                    # Comprehensive health check
# nself status                    # View service status
# nself logs                      # Check service logs
# nself resources                 # Check resource usage

# Alternative: Use Docker commands directly
docker ps                         # View running containers
docker logs [container-name]      # View container logs
docker stats                      # Monitor resource usage

Installation Issues

nself CLI Won't Install

Problem

Installation script fails or CLI command not found after installation.

Solutions

# Check if installation directory exists
ls -la ~/.nself/bin/

# Add to PATH manually
echo 'export PATH="$HOME/.nself/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

# Or use direct path
/home/$USER/.nself/bin/nself version

# Alternative: Install manually
wget https://github.com/acamarata/nself/releases/latest/download/nself-linux-amd64.tar.gz
tar -xzf nself-linux-amd64.tar.gz
sudo mv nself /usr/local/bin/

Permission Denied Errors

Problem

Getting permission errors when running nself commands.

Solutions

# Add user to docker group
sudo usermod -aG docker $USER
newgrp docker

# Fix file permissions
sudo chown -R $USER:$USER ~/.nself/
chmod +x ~/.nself/bin/nself

# Verify Docker permissions
docker run hello-world

# If still having issues, restart Docker
sudo systemctl restart docker

Docker Issues

Docker Not Running

Problem

"Cannot connect to the Docker daemon" error.

Solutions

# Check if Docker is running
sudo systemctl status docker

# Start Docker service
sudo systemctl start docker

# Enable Docker to start on boot
sudo systemctl enable docker

# On macOS, start Docker Desktop
open -a Docker

# Test Docker installation
docker --version
docker-compose --version

Port Already in Use

Problem

Ports 5432, 8080, or other default ports are already in use.

Solutions

# Find what's using the port
sudo netstat -tulpn | grep :5432
sudo lsof -i :8080

# Kill the process using the port
sudo kill -9 [PID]

# Or change ports in configuration
nself config set POSTGRES_PORT 5433
nself config set HASURA_PORT 8081

# Rebuild configuration
nself build && nself up

Out of Disk Space

Problem

Docker containers fail to start due to insufficient disk space.

Solutions

# Check disk usage
df -h
docker system df

# Clean up Docker resources
docker system prune -a --volumes

# Remove unused images
docker image prune -a

# Remove stopped containers
docker container prune

# Remove unused volumes
docker volume prune

# Clean up nself resources
nself cleanup

Service-Specific Issues

PostgreSQL Won't Start

Problem

PostgreSQL container fails to start or immediately exits.

Solutions

# Check PostgreSQL logs
nself logs postgres

# Common issues and fixes:

# 1. Corrupted data directory
nself down
docker volume rm $(docker volume ls -q | grep postgres)
nself up

# 2. Wrong permissions
nself exec postgres chown -R postgres:postgres /var/lib/postgresql/data

# 3. Invalid configuration
nself config validate
nself config get POSTGRES_PASSWORD

# 4. Port conflict
nself config set POSTGRES_PORT 5433
nself build && nself up

Hasura GraphQL Engine Issues

Problem

Hasura console not accessible or GraphQL API returning errors.

Solutions

# Check Hasura logs
nself logs hasura

# Common issues:

# 1. Database connection failed
nself exec hasura hasura-cli console --project-dir /app
nself config get HASURA_GRAPHQL_DATABASE_URL

# 2. Invalid admin secret
nself config get HASURA_GRAPHQL_ADMIN_SECRET
curl -H "X-Hasura-Admin-Secret: your-secret" http://localhost:8080/v1/query

# 3. Migration issues
nself exec hasura hasura-cli migrate status
nself exec hasura hasura-cli migrate apply

# 4. Metadata issues
nself exec hasura hasura-cli metadata reload

MinIO Storage Issues

Problem

Cannot access MinIO console or API returning authentication errors.

Solutions

# Check MinIO logs
nself logs minio

# Test MinIO health
curl http://localhost:9000/minio/health/live

# Check credentials
nself config get MINIO_ROOT_USER
nself config get MINIO_ROOT_PASSWORD

# Reset MinIO data (⚠️ destroys all stored files)
nself down
docker volume rm $(docker volume ls -q | grep minio)
nself up

# Check bucket permissions
nself exec minio mc ls local/
nself exec minio mc policy get local/your-bucket

Redis Connection Issues

Problem

Applications cannot connect to Redis or Redis commands timing out.

Solutions

# Check Redis status
nself logs redis
nself exec redis redis-cli ping

# Test Redis connection
nself exec redis redis-cli -h redis -p 6379 ping

# Check Redis configuration
nself exec redis redis-cli config get maxmemory
nself exec redis redis-cli info memory

# Clear Redis data if corrupted
nself exec redis redis-cli flushall

# Check for memory issues
nself exec redis redis-cli info stats | grep rejected_connections

Configuration Issues

Environment Variables Not Loading

Problem

Environment variables from .env files are not being loaded by services.

Solutions

# Verify .env file exists and is readable
ls -la .env.local
cat .env.local

# Check for syntax errors in .env file
nself config validate

# Rebuild configuration
nself build

# Check if variables are properly set in containers
nself exec postgres env | grep POSTGRES
nself exec hasura env | grep HASURA

# Remove and recreate containers
nself down
nself up --recreate

SSL/TLS Certificate Issues

Problem

SSL certificate errors or HTTPS not working properly.

Solutions

# Check certificate files exist
ls -la /etc/ssl/certs/
ls -la ~/.nself/ssl/

# Verify certificate validity
openssl x509 -in cert.pem -text -noout
openssl x509 -in cert.pem -enddate -noout

# Check nginx SSL configuration
nself exec nginx nginx -t
nself logs nginx | grep ssl

# Regenerate certificates for development
rm -rf ~/.nself/ssl/
nself build && nself up

Performance Issues

Slow Database Queries

Problem

Database queries are taking too long to execute.

Solutions

# Check slow queries
nself exec postgres psql -U postgres -c "SELECT * FROM pg_stat_statements ORDER BY total_time DESC LIMIT 10;"

# Analyze query performance
nself exec postgres psql -U postgres -c "EXPLAIN ANALYZE SELECT * FROM your_table;"

# Check for missing indexes
nself exec postgres psql -U postgres -c "SELECT tablename, attname, n_distinct, correlation FROM pg_stats;"

# Monitor active queries
nself exec postgres psql -U postgres -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query FROM pg_stat_activity WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes';"

# Check database statistics
nself exec postgres psql -U postgres -c "SELECT * FROM pg_stat_database;"

# Vacuum and analyze tables
nself exec postgres psql -U postgres -c "VACUUM ANALYZE;"

High Memory Usage

Problem

Services consuming excessive memory or system running out of memory.

Solutions

# Check memory usage by service
nself resources
docker stats

# Set memory limits
nself config set POSTGRES_MEMORY_LIMIT 1024MB
nself config set HASURA_MEMORY_LIMIT 512MB
nself config set REDIS_MEMORY_LIMIT 256MB

# Check for memory leaks
nself exec postgres ps aux --sort=-%mem | head
nself logs postgres | grep -i memory

# Tune PostgreSQL memory settings
nself exec postgres psql -U postgres -c "SHOW shared_buffers;"
nself exec postgres psql -U postgres -c "SHOW work_mem;"

# Clear Redis memory
nself exec redis redis-cli flushall

Network and Connectivity Issues

Services Cannot Communicate

Problem

Services cannot reach each other (e.g., Hasura cannot connect to PostgreSQL).

Solutions

# Check Docker networks
docker network ls
docker network inspect nself_default

# Test connectivity between services
nself exec hasura ping postgres
nself exec hasura nc -zv postgres 5432

# Check service hostnames and ports
nself exec hasura nslookup postgres
nself exec hasura cat /etc/hosts

# Verify services are on the same network
docker inspect hasura | grep NetworkMode
docker inspect postgres | grep NetworkMode

# Recreate network
nself down
docker network rm nself_default
nself up

Cannot Access Services from Host

Problem

Cannot access services like Hasura console from your browser.

Solutions

# Check if ports are properly mapped
nself ports
docker ps

# Test if service is responding
curl http://localhost:8080/healthz
curl http://localhost:9000/minio/health/live

# Check firewall settings
sudo ufw status
sudo iptables -L

# For remote access, check if binding to all interfaces
nself config get HASURA_ADDRESS
nself config set HASURA_ADDRESS 0.0.0.0:8080

# Verify port is not blocked
telnet localhost 8080
nc -zv localhost 8080

Data Loss and Corruption

Database Data Lost

Problem

Database data has disappeared or been corrupted.

Solutions

# Check if volume still exists
docker volume ls | grep postgres
docker volume inspect your-postgres-volume

# Check database status
nself exec postgres psql -U postgres -c "\l"
nself exec postgres psql -U postgres -c "\dt"

# Restore from backup if available
nself db restore --file backup_20240101.sql

# Check for backup files
ls -la ./backups/
nself backup list

# If no backups, check Docker volume backup
docker run --rm -v your-postgres-volume:/data -v $(pwd):/backup alpine ls -la /data

# Recovery from corrupted database
nself exec postgres pg_resetwal -f /var/lib/postgresql/data
nself restart postgres

Debugging Techniques

Interactive Debugging

# Access service shells for debugging
nself exec -it postgres bash
nself exec -it hasura sh
nself exec -it redis sh

# Monitor real-time logs
nself logs -f --tail 100

# Watch resource usage
watch -n 2 'docker stats --no-stream'

# Network debugging
nself exec hasura netstat -tlpn
nself exec postgres ss -tlpn

Log Analysis

# Extract specific error patterns
nself logs | grep -i error
nself logs postgres | grep -i "could not connect"
nself logs hasura | grep -i "database"

# Save logs for analysis
nself logs --since "1h" > debug_logs.txt

# Monitor specific service
nself logs -f hasura | grep -v "GET /healthz"

Recovery Procedures

Emergency Recovery

# Complete system reset (⚠️ destroys all data)
nself emergency-reset

# Soft reset (preserves database data)
nself reset --soft

# Partial recovery
nself down postgres hasura
nself up postgres
# Wait for database to start
nself up hasura

# Configuration recovery
cp .env.example .env.local
nself config generate
nself build && nself up

Backup and Restore

# Create emergency backup
nself db backup --name emergency_$(date +%Y%m%d_%H%M%S)

# Restore from specific backup
nself db restore --name emergency_20240101_120000

# Export all data
nself db dump --all > full_backup.sql

# Import all data
nself db restore --file full_backup.sql

Prevention Strategies

Regular Maintenance

Regular Updates: Keep nself and Docker images updated
Automated Backups: Set up daily database backups
Monitor Resources: Check disk space and memory regularly
Log Rotation: Prevent log files from filling disk
Health Checks: Run periodic health checks

Monitoring Setup

# Set up monitoring script
#!/bin/bash
# check_nself.sh
nself doctor
if [ $? -ne 0 ]; then
    echo "nself health check failed" | mail -s "nself Alert" admin@example.com
fi

# Add to crontab
crontab -e
# Add: */15 * * * * /path/to/check_nself.sh

Getting Help

Collecting Debug Information

# Generate comprehensive debug report
nself debug-report

# Manual debug info collection
echo "=== System Info ===" > debug_info.txt
uname -a >> debug_info.txt
docker --version >> debug_info.txt
nself version >> debug_info.txt

echo "=== Configuration ===" >> debug_info.txt
nself config show >> debug_info.txt

echo "=== Service Status ===" >> debug_info.txt
nself status --verbose >> debug_info.txt

echo "=== Recent Logs ===" >> debug_info.txt
nself logs --tail 200 >> debug_info.txt

Community Support

GitHub Issues: Report bugs and request features
Discussions: Community discussions and Q&A
Documentation: Check other documentation pages for specific topics
Examples: Look at example configurations in the repository

Still Need Help?

If this guide doesn't solve your issue, please create a GitHub issue with the output from nself debug-report and a detailed description of the problem.

Next Steps

After resolving issues:

Monitoring - Set up monitoring to prevent future issues
Backup & Restore - Implement backup strategies
Performance - Optimize your setup
Support - Learn about getting ongoing support

Most issues can be resolved with the solutions in this guide. Remember to always backup your data before making major changes and monitor your system regularly to prevent problems.