Merge #15: feat: [#14] Phase 1 - 12-Factor App configuration management system

77c3fa3 docs: [#14] add comprehensive integration testing guide execution instructions (Jose Celano)
613b5df docs: [#14] add application storage cleanup step to integration testing guide (Jose Celano)
cc87760 fix: nginx template variable escaping and add documentation (Jose Celano)
f09efaf refactor: [#14] move configuration files to infrastructure layer (Jose Celano)
a8b11c5 refactor: move nginx.conf to infrastructure templates and improve configuration management (Jose Celano)
10592d0 feat: [#14] Complete integration test refactoring and verification (Jose Celano)
a0456e8 refactor: [#14] align integration tests with 12-factor configuration approach (Jose Celano)
00e13bf docs: update integration testing guide with comprehensive smoke tests and fixes (Jose Celano)
9bbd7bf docs: [#14] update integration testing guide with network architecture and proper testing procedures (Jose Celano)
458546a fix: [#14] resolve tracker database driver configuration issue (Jose Celano)
0c9d081 docs: [#integration-tests] add OpenTofu state refresh step to prevent IP detection issues (Jose Celano)
59bdd15 docs: improve integration testing guide with directory indicators and new config workflow (Jose Celano)
83d62f0 docs: add security documentation for plain text secrets (Jose Celano)
770fe25 fix: correct environment variable names and database config (Jose Celano)
2d19b12 refactor: simplify environment variables to secrets-only approach (Jose Celano)
6b46ddd fix: [#14] resolve yamllint validation warnings in config scripts (Jose Celano)
94d278b feat: [#14] implement Phase 1 of 12-Factor App refactor - configuration management (Jose Celano)

Pull request description:

  ## Phase 1: Configuration Management System

  This PR implements the first phase of the 12-Factor App refactoring for the Torrust Tracker Demo, focusing on configuration management and environment-based deployment.

  ### 🎯 **Objectives Completed**

  - ✅ **Environment-based configuration** using `.env` files and templates
  - ✅ **Configuration templates** with `envsubst` for dynamic generation
  - ✅ **Automated configuration scripts** with validation and colored output
  - ✅ **Docker Compose integration** with generated environment files
  - ✅ **Makefile automation** for configuration and deployment workflows
  - ✅ **MySQL database migration** from SQLite for production readiness

  ### 🏗️ **Architecture Changes**

  #### Configuration Structure
  ```
  infrastructure/config/
  ├── environments/
  │   ├── local.env          # Local development configuration
  │   └── production.env     # Production deployment configuration
  └── templates/
      ├── tracker.toml.tpl         # Torrust Tracker configuration template
      ├── prometheus.yml.tpl       # Prometheus monitoring template
      └── docker-compose.env.tpl   # Docker Compose environment template
  ```

  #### New Automation Scripts
  - **`configure-env.sh`** - Processes templates and generates configuration files
  - **`validate-config.sh`** - Validates generated configurations for syntax and completeness

  #### Enhanced Makefile Targets
  - `configure-local` / `configure-production` - Generate environment-specific configs
  - `validate-config` / `validate-config-production` - Validate configurations
  - `deploy-local` / `deploy-production` - Full deployment workflows

  ### 🔧 **Key Features**

  #### 1. **Template-Based Configuration**
  All configuration files are now generated from templates using environment variables:
  - No more hardcoded values in configuration files
  - Environment-specific settings (logging levels, database URLs, etc.)
  - Secrets management through environment variables

  #### 2. **Environment Separation**
  - **Local**: Debug logging, direct access, SQLite/MySQL support
  - **Production**: Info logging, reverse proxy mode, MySQL database, SSL ready

  #### 3. **Validation & Quality Assurance**
  - TOML and YAML syntax validation
  - Environment-specific configuration checks
  - Template variable substitution verification
  - Linting integration with project standards

  #### 4. **Docker Compose Integration**
  - Generated `.env` files for Docker Compose
  - Consistent variable naming across all services
  - Environment-specific service configurations

  ### 🚀 **Usage**

  #### Quick Start - Local Development
  ```bash
  make configure-local    # Generate local configurations
  make validate-config    # Validate configurations
  make deploy-local       # Deploy with local settings
  ```

  #### Production Deployment
  ```bash
  # Set required secrets as environment variables
  export TORRUST_PROD_DATABASE_URL="mysql://user:pass@host:3306/db"
  export TORRUST_PROD_API_TOKEN="your-api-token"
  # ... other production secrets

  make configure-production  # Generate production configurations
  make validate-config-production  # Validate configurations
  make deploy-production    # Deploy to production
  ```

  ### 🛡️ **Security Improvements**

  - **No hardcoded secrets** - All sensitive values use environment variables
  - **Environment isolation** - Clear separation between local and production configs
  - **Validation safeguards** - Prevent deployment with misconfigured settings

  ### 📋 **Breaking Changes**

  - Configuration files are now generated and should not be edited directly
  - `.env` files must be generated using the new scripts
  - Database configuration now defaults to MySQL for both environments

  ### 🧪 **Testing**

  All changes have been validated with:
  - ✅ **Linting** - yamllint, shellcheck, markdownlint
  - ✅ **Configuration validation** - Both local and production environments
  - ✅ **Template processing** - All templates generate correctly
  - ✅ **Docker Compose integration** - Services start with generated configs

  ### 📚 **Documentation**

  Updated documentation includes:
  - Phase 1 implementation guide with detailed migration steps
  - Configuration management workflow documentation
  - Makefile target reference and usage examples

  ### 🔄 **Next Steps (Future PRs)**

  - **Phase 2**: Infrastructure as Code enhancements
  - **Phase 3**: Service isolation and containerization improvements
  - **Phase 4**: Monitoring and observability integrations

  ---

  **Ready for Review**: This PR represents a complete Phase 1 implementation of the 12-Factor App methodology. All existing functionality is preserved while adding robust configuration management capabilities.

ACKs for top commit:
  josecelano:
    ACK 77c3fa3

Tree-SHA512: 359047b1b8d34aa9586e26011c6ab33fd9d28a5a3f931ff9eef3d1d67c7ee2680f8feaf9231ea761a6d33987f0e6695fd3611130144ae5cf7b4a2618bbf5104d

Author: josecelano

.github/prompts/run-integration-testing-guide.prompt.md

@@ -0,0 +1,153 @@
+---
+mode: agent
+---
+
+# Integration Testing Guide Execution Instructions
+
+As an expert system administrator, you will execute the **complete integration testing process** following the [Integration Testing Guide](../../docs/guides/integration-testing-guide.md).
+
+## 📋 Overview
+
+This guide performs a **full end-to-end integration test** that includes:
+
+1. **Clean existing state** (VM, application data, certificates)
+2. **Deploy fresh infrastructure** (VM with Ubuntu 24.04)
+3. **Wait for cloud-init completion** (system provisioning)
+4. **Run comprehensive integration tests** (services, connectivity, functionality)
+5. **Perform smoke testing** (external validation with official client tools)
+6. **Clean up resources** (return to clean state)
+
+**Expected Duration**: ~8-12 minutes total
+**Prerequisites**: Must have completed initial setup (`make test-prereq`)
+
+## 🎯 Execution Requirements
+
+### CRITICAL Rules to Follow:
+
+1. **Sequential Execution**: Follow steps in exact order - do NOT skip or reorder
+2. **No Command Modifications**: Execute commands exactly as written in the guide
+3. **Working Directory**: Always run from project root directory
+4. **Error Handling**: Document any failures or deviations immediately
+5. **Complete Process**: Execute the entire guide from start to finish
+
+### What Gets Cleaned (Destructive Operations):
+
+- **Virtual Machine**: Complete VM destruction and recreation
+- **Application Storage**: Database, SSL certificates, configuration files
+- **OpenTofu State**: Infrastructure state reset
+- **libvirt Resources**: VM disks, cloud-init ISOs, network configurations
+
+## 📝 Step-by-Step Instructions
+
+### Phase 1: Preparation and Cleanup
+
+- **Step 1.1-1.8**: Clean existing infrastructure and application state
+- **Critical**: Step 1.8 (Clean Application Storage) is destructive but recommended
+- **Outcome**: Clean slate for fresh deployment
+
+### Phase 2: Infrastructure Deployment
+
+- **Step 2.1-2.4**: Deploy VM with OpenTofu/Terraform
+- **Critical**: Wait for cloud-init completion (Step 3)
+- **Outcome**: Provisioned VM with Torrust Tracker ready
+
+### Phase 3: Integration Testing
+
+- **Step 4**: Run comprehensive integration tests
+- **Step 5**: Optional manual verification
+- **Step 6**: Optional performance testing
+- **Outcome**: Validated working system
+
+### Phase 4: External Validation
+
+- **Step 7**: External smoke testing with official client tools
+- **Reference**: Use [Smoke Testing Guide](../../docs/guides/smoke-testing-guide.md) for details
+- **Outcome**: Black-box validation of tracker functionality
+
+### Phase 5: Cleanup
+
+- **Step 8**: Clean up all resources
+- **Step 9**: Review insights and best practices
+- **Outcome**: Return to clean state
+
+## 🚨 Important Notes
+
+### SSH Key Configuration
+
+- **Required**: Must configure SSH keys before deployment
+- **Location**: `infrastructure/terraform/local.tfvars`
+- **Template**: Available in `infrastructure/terraform/terraform.tfvars.example`
+
+### Cloud-Init Wait Time
+
+- **Critical**: DO NOT skip Step 3 (cloud-init completion)
+- **Duration**: 2-3 minutes typically
+- **Failure Mode**: SSH connection failures if rushed
+
+### Error Documentation
+
+- **Immediate**: Document any command failures or unexpected outputs
+- **Location**: Add issues directly to the integration testing guide
+- **Format**: Include error messages, context, and resolution steps
+
+### Non-Standard Commands
+
+- **Approval Required**: Only execute commands not in the guide if absolutely necessary
+- **Documentation**: Clearly indicate when deviating from guide
+- **Justification**: Explain why the deviation was needed
+
+## 🔧 Troubleshooting Guidance
+
+### Common Issues and Solutions:
+
+1. **"Command not found"**: Verify you're in project root directory
+2. **SSH connection failures**: Ensure cloud-init has completed
+3. **libvirt permission errors**: Check user is in libvirt group
+4. **VM deployment timeouts**: Normal during cloud-init, wait longer
+5. **Storage volume conflicts**: Run manual cleanup steps from guide
+
+### When to Deviate from Guide:
+
+- **System-specific issues**: Different Linux distributions may need adjustments
+- **Network configuration**: Firewall or DNS issues requiring resolution
+- **Permission problems**: User/group configuration fixes
+- **Always document**: Any deviations with full explanation
+
+## 📊 Success Criteria
+
+### Integration Test Success Indicators:
+
+- ✅ All services start successfully (Docker Compose)
+- ✅ Tracker responds to UDP/HTTP requests
+- ✅ API endpoints return expected data
+- ✅ Grafana dashboards display metrics
+- ✅ MySQL database is accessible and functional
+
+### Smoke Test Success Indicators:
+
+- ✅ UDP tracker clients receive responses
+- ✅ HTTP tracker clients receive responses
+- ✅ API health checks return "Ok"
+- ✅ Statistics endpoints return valid data
+- ✅ Metrics endpoints return Prometheus data
+
+## 🎯 Final Deliverables
+
+Upon completion, you should have:
+
+1. **Executed Complete Guide**: All steps from 1.1 through 9
+2. **Documented Issues**: Any problems encountered and how they were resolved
+3. **Validated Functionality**: Both integration and smoke tests passed
+4. **Clean State**: All resources cleaned up and ready for next test
+5. **Updated Documentation**: Any guide improvements or corrections needed
+
+## 📖 Additional Resources
+
+- **Integration Testing Guide**: [docs/guides/integration-testing-guide.md](../../docs/guides/integration-testing-guide.md)
+- **Smoke Testing Guide**: [docs/guides/smoke-testing-guide.md](../../docs/guides/smoke-testing-guide.md)
+- **Quick Start Guide**: [docs/infrastructure/quick-start.md](../../docs/infrastructure/quick-start.md)
+- **Troubleshooting**: See infrastructure documentation for libvirt and OpenTofu issues
+
+---
+
+**Remember**: This is a comprehensive test that validates the entire deployment pipeline. Take your time, follow each step carefully, and document everything for future improvements.

.yamllint-ci.yml

@@ -1,5 +1,8 @@
 extends: default
 
+ignore: |
+  application/storage/
+
 rules:
   line-length:
     max: 120 # More reasonable for infrastructure code

Makefile

@@ -1,5 +1,5 @@
 # Makefile for Torrust Tracker Local Testing Infrastructure
-.PHONY: help init plan apply destroy test clean status refresh-state ssh install-deps console vm-console lint lint-yaml lint-shell lint-markdown
+.PHONY: help init plan apply destroy test clean status refresh-state ssh install-deps console vm-console lint lint-yaml lint-shell lint-markdown configure-local configure-production validate-config validate-config-production deploy-local deploy-production start-services stop-services
 
 # Default variables
 VM_NAME ?= torrust-tracker-demo
@@ -189,7 +189,23 @@ clean-and-fix: ## Clean up all VMs and fix libvirt permissions
 	@cd $(TERRAFORM_DIR) && rm -f terraform.tfstate terraform.tfstate.backup .terraform.lock.hcl 2>/dev/null || true
 	@echo "3. Cleaning libvirt images:"
 	@sudo rm -f /var/lib/libvirt/images/torrust-tracker-demo* /var/lib/libvirt/images/ubuntu-24.04-base.qcow2 2>/dev/null || true
-	@echo "4. Fixing libvirt setup:"
+	@echo "4. Cleaning application storage (generated configuration files):"
+	@if [ -d "application/storage" ]; then \
+		echo "   WARNING: This will delete all generated configuration files in application/storage/"; \
+		echo "   This includes nginx configs, tracker configs, and any cached data."; \
+		echo "   These files will be regenerated when you run 'make configure-local'."; \
+		read -p "   Do you want to delete application/storage? (y/N): " confirm; \
+		if [ "$$confirm" = "y" ] || [ "$$confirm" = "Y" ]; then \
+			echo "   Removing application/storage..."; \
+			rm -rf application/storage; \
+			echo "   ✓ Application storage cleaned"; \
+		else \
+			echo "   Skipping application/storage cleanup"; \
+		fi; \
+	else \
+		echo "   No application/storage directory found"; \
+	fi
+	@echo "5. Fixing libvirt setup:"
 	@$(MAKE) fix-libvirt
 	@echo "✓ Clean up complete. You can now run 'make apply' safely."
 
@@ -342,3 +358,61 @@ vm-console: ## Access VM graphical console (GUI)
 		echo "virt-viewer not found. Please install it:"; \
 		echo "  sudo apt install virt-viewer"; \
 	fi
+
+# Configuration Management Targets
+configure-local: ## Generate local environment configuration
+	@echo "Generating local environment configuration..."
+	@infrastructure/scripts/configure-env.sh local
+
+configure-production: ## Generate production environment configuration (requires secrets)
+	@echo "Generating production environment configuration..."
+	@infrastructure/scripts/configure-env.sh production
+
+validate-config: ## Validate generated configuration files
+	@echo "Validating configuration files..."
+	@infrastructure/scripts/validate-config.sh local
+
+validate-config-production: ## Validate production configuration files
+	@echo "Validating production configuration files..."
+	@infrastructure/scripts/validate-config.sh production
+
+# Deployment workflow targets
+deploy-local: configure-local ## Deploy VM and configure for local environment
+	@echo "Deploying local environment..."
+	@$(MAKE) apply
+	@echo "Waiting for VM to be ready..."
+	@sleep 30
+	@echo "Starting application services..."
+	@$(MAKE) start-services
+
+deploy-production: configure-production ## Deploy and configure for production environment (requires secrets)
+	@echo "Deploying production environment..."
+	@$(MAKE) apply
+	@echo "Waiting for VM to be ready..."
+	@sleep 30
+	@echo "Starting application services..."
+	@$(MAKE) start-services
+
+start-services: ## Start Docker Compose services in the VM
+	@echo "Starting Docker Compose services..."
+	@VM_IP=$$(cd $(TERRAFORM_DIR) && tofu output -raw vm_ip 2>/dev/null) || \
+	 VM_IP=$$(virsh domifaddr $(VM_NAME) | grep ipv4 | awk '{print $$4}' | cut -d'/' -f1); \
+	if [ -n "$$VM_IP" ]; then \
+		echo "Starting services on $$VM_IP..."; \
+		ssh -o StrictHostKeyChecking=no torrust@$$VM_IP 'cd /home/torrust/github/torrust/torrust-tracker-demo/application && docker compose up -d'; \
+	else \
+		echo "Could not get VM IP. Is the VM deployed?"; \
+		exit 1; \
+	fi
+
+stop-services: ## Stop Docker Compose services in the VM
+	@echo "Stopping Docker Compose services..."
+	@VM_IP=$$(cd $(TERRAFORM_DIR) && tofu output -raw vm_ip 2>/dev/null) || \
+	 VM_IP=$$(virsh domifaddr $(VM_NAME) | grep ipv4 | awk '{print $$4}' | cut -d'/' -f1); \
+	if [ -n "$$VM_IP" ]; then \
+		echo "Stopping services on $$VM_IP..."; \
+		ssh -o StrictHostKeyChecking=no torrust@$$VM_IP 'cd /home/torrust/github/torrust/torrust-tracker-demo/application && docker compose down'; \
+	else \
+		echo "Could not get VM IP. Is the VM deployed?"; \
+		exit 1; \
+	fi

application/.env.production

@@ -108,11 +108,12 @@ services:
     tty: true
     restart: unless-stopped
     environment:
-      - USER_ID=${USER_ID}
-      - TORRUST_TRACKER_DATABASE=${TORRUST_TRACKER_DATABASE:-mysql}
-      - TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER=${TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER:-mysql}
-      - TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__PATH=${TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__PATH:-mysql://torrust:password@mysql:3306/torrust_tracker}
-      - TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN=${TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN:-MyAccessToken}
+      - USER_ID=${USER_ID:-1000}
+      # Database connection for tracker (using Figment override pattern)
+      - TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER=mysql
+      - TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__PATH=mysql://${MYSQL_USER}:${MYSQL_PASSWORD}@mysql:3306/${MYSQL_DATABASE}
+      # Admin API token for tracker (using Figment override pattern)
+      - TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN=${TRACKER_ADMIN_TOKEN}
     networks:
       - backend_network
     ports:

application/compose.yaml

@@ -1,21 +1,42 @@
 #!/bin/bash
 
+# Torrust Tracker Demo Installation Script
+# This script creates the required directory structure for the application.
+# Following 12-factor principles, it expects .env to be provided by the infrastructure layer.
+
 if ! [ -f "./.env" ]; then
-	echo "Creating compose .env './.env'"
-	cp .env.production .env
+	echo "ERROR: Environment file './.env' not found!"
+	echo "The .env file must be provided by the infrastructure configuration system."
+	echo "Expected location: $(pwd)/.env"
+	echo ""
+	echo "To generate the .env file, run:"
+	echo "  make configure-local    # For local development"
+	echo "  make configure-production # For production deployment"
+	exit 1
 fi
 
+echo "Found environment file: ./.env"
+
 ## Proxy
 
 mkdir -p ./storage/proxy/etc/nginx-conf
 mkdir -p ./storage/proxy/webroot
 mkdir -p ./storage/dhparam
 
+# Verify nginx configuration exists (should be provided by infrastructure)
 if ! [ -f "./storage/proxy/etc/nginx-conf/nginx.conf" ]; then
-	echo "Creating proxy config file: './storage/proxy/etc/nginx-conf/nginx.conf'"
-	cp ./share/container/default/config/nginx.conf ./storage/proxy/etc/nginx-conf/nginx.conf
+	echo "ERROR: Nginx configuration file './storage/proxy/etc/nginx-conf/nginx.conf' not found!"
+	echo "This file should be generated by the infrastructure configuration system."
+	echo "Expected location: $(pwd)/storage/proxy/etc/nginx-conf/nginx.conf"
+	echo ""
+	echo "To generate the configuration file, run:"
+	echo "  make configure-local    # For local development"
+	echo "  make configure-production # For production deployment"
+	exit 1
 fi
 
+echo "Found nginx configuration: ./storage/proxy/etc/nginx-conf/nginx.conf"
+
 ## Certbot
 
 mkdir -p ./storage/certbot/etc
@@ -33,16 +54,37 @@ fi
 
 mkdir -p ./storage/tracker/etc
 
-if ! [ -f "./storage/tracker/etc/tracker.prod.container.sqlite3.toml" ]; then
-	echo "Creating tracker configuration: './storage/tracker/etc/tracker.toml'"
-	cp ./share/container/default/config/tracker.prod.container.sqlite3.toml ./storage/tracker/etc/tracker.toml
+# Verify tracker configuration exists (should be provided by infrastructure)
+if ! [ -f "./storage/tracker/etc/tracker.toml" ]; then
+	echo "ERROR: Tracker configuration file './storage/tracker/etc/tracker.toml' not found!"
+	echo "This file should be generated by the infrastructure configuration system."
+	echo "Expected location: $(pwd)/storage/tracker/etc/tracker.toml"
+	echo ""
+	echo "To generate the configuration file, run:"
+	echo "  make configure-local    # For local development"
+	echo "  make configure-production # For production deployment"
+	exit 1
 fi
 
+echo "Found tracker configuration: ./storage/tracker/etc/tracker.toml"
+
 ## Prometheus
 
 mkdir -p ./storage/prometheus/etc
 
+# Verify prometheus configuration exists (should be provided by infrastructure)
 if ! [ -f "./storage/prometheus/etc/prometheus.yml" ]; then
-	echo "Creating prometheus config file: './storage/prometheus/etc/prometheus.yml'"
-	cp ./share/container/default/config/prometheus.yml ./storage/prometheus/etc/prometheus.yml
+	echo "ERROR: Prometheus configuration file './storage/prometheus/etc/prometheus.yml' not found!"
+	echo "This file should be generated by the infrastructure configuration system."
+	echo "Expected location: $(pwd)/storage/prometheus/etc/prometheus.yml"
+	echo ""
+	echo "To generate the configuration file, run:"
+	echo "  make configure-local    # For local development"
+	echo "  make configure-production # For production deployment"
+	exit 1
 fi
+
+echo "Found prometheus configuration: ./storage/prometheus/etc/prometheus.yml"
+
+echo "Installation completed successfully!"
+echo "All required directories created and configuration files verified."