(feat) Integrate Secret Manager

[pacphi]

Scaleway Secret Manager Integration for Coder Infrastructure

Executive Summary

This document proposes implementing Scaleway Secret Manager with External Secrets Operator to address critical security vulnerabilities identified in the Terraform Security Review. The solution provides enterprise-grade secret management for Coder workspaces while eliminating credential exposure in Terraform state files and CI/CD pipelines.

Current Security Vulnerabilities

Critical Issues Identified

• 🚨 CRITICAL: Database credentials stored in plain text in Terraform outputs
• 🚨 CRITICAL: Sensitive values exposed in Terraform state files
• 🔴 HIGH: API keys and tokens managed as Kubernetes secrets without encryption at rest
• 🔴 HIGH: No centralized secret lifecycle management
• 🔴 HIGH: Workspace templates accessing secrets through environment variables

Proposed Solution: Scaleway Secret Manager Integration

Architecture Overview

┌─────────────────────┐    ┌──────────────────────┐    ┌─────────────────────┐
│   Scaleway Secret   │    │  External Secrets    │    │   Kubernetes        │
│     Manager         │◄───┤     Operator         ├───►│     Secrets         │
│                     │    │                      │    │                     │
│  - AES-256 KMS      │    │  - Auto-sync         │    │  - Workspace Pods   │
│  - Regional Deploy  │    │  - Version Control   │    │  - Coder Templates  │
│  - API Access       │    │  - RBAC Integration  │    │  - Env Variables    │
└─────────────────────┘    └──────────────────────┘    └─────────────────────┘
                                      │
                                      ▼
                           ┌──────────────────────┐
                           │   Coder Workspaces   │
                           │                      │
                           │  - Secure DB Access  │
                           │  - API Key Injection │
                           │  - Template Secrets  │
                           └──────────────────────┘

Core Components

1. Scaleway Secret Manager: Centralized, encrypted secret storage
2. External Secrets Operator: Kubernetes-native secret synchronization
3. Secret Store Configuration: Environment-specific secret access policies
4. Coder Template Integration: Secure secret injection into workspaces

Implementation Plan

Phase 1: Infrastructure Setup (Week 1)

Day 1-2: Scaleway Secret Manager Deployment

Owner: Infrastructure Team
Priority: P0

1. Create Secret Manager Module

   # File: modules/secrets-manager/main.tf
   resource "scaleway_secret" "database_credentials" {
     name        = "${var.environment}-database-credentials"
     description = "Database connection credentials for ${var.environment}"
     region      = var.region
     project_id  = var.project_id
   
     tags = {
       Environment = var.environment
       Service     = "coder"
       ManagedBy   = "terraform"
     }
   }
   
   resource "scaleway_secret_version" "database_password" {
     secret_id = scaleway_secret.database_credentials.id
     data      = jsonencode({
       username = var.database_username
       password = var.database_password
       host     = var.database_host
       port     = var.database_port
       database = var.database_name
       ssl_mode = "require"
     })
   }
   
   resource "scaleway_secret" "coder_admin_credentials" {
     name        = "${var.environment}-coder-admin"
     description = "Coder admin user credentials for ${var.environment}"
     region      = var.region
     project_id  = var.project_id
   
     tags = {
       Environment = var.environment
       Service     = "coder"
       Type        = "admin"
       ManagedBy   = "terraform"
     }
   }
   
   resource "scaleway_secret_version" "coder_admin_password" {
     secret_id = scaleway_secret.coder_admin_credentials.id
     data      = jsonencode({
       username = "admin"
       password = random_password.coder_admin.result
       email    = var.admin_email
     })
   }
   
   resource "random_password" "coder_admin" {
     length  = 32
     special = true
   }

2. IAM Configuration for Secret Access

   # File: modules/secrets-manager/iam.tf
   resource "scaleway_iam_application" "external_secrets" {
     name        = "external-secrets-${var.environment}"
     description = "Application for External Secrets Operator in ${var.environment}"
   }
   
   resource "scaleway_iam_api_key" "external_secrets" {
     application_id = scaleway_iam_application.external_secrets.id
     description    = "API key for External Secrets Operator"
   }
   
   resource "scaleway_iam_policy" "secret_read" {
     name           = "secret-read-${var.environment}"
     description    = "Allow read access to secrets"
     organization_id = var.organization_id
   
     rule {
       project_ids          = [var.project_id]
       permission_set_names = ["SecretManagerReadOnly"]
     }
   }
   
   resource "scaleway_iam_group_membership" "external_secrets" {
     group_id       = scaleway_iam_group.secret_readers.id
     application_id = scaleway_iam_application.external_secrets.id
   }

Day 3-4: External Secrets Operator Deployment

Owner: Platform Team
Priority: P0

1. Deploy External Secrets Operator

   # File: modules/external-secrets/main.tf
   resource "helm_release" "external_secrets" {
     name       = "external-secrets"
     repository = "https://charts.external-secrets.io"
     chart      = "external-secrets"
     version    = "0.9.11"
     namespace  = "external-secrets-system"
     create_namespace = true
   
     values = [
       yamlencode({
         installCRDs = true
         replicaCount = 2
   
         resources = {
           limits = {
             cpu    = "100m"
             memory = "128Mi"
           }
           requests = {
             cpu    = "50m"
             memory = "64Mi"
           }
         }
   
         securityContext = {
           runAsNonRoot = true
           runAsUser    = 65534
         }
   
         podSecurityContext = {
           fsGroup = 65534
         }
       })
     ]
   
     depends_on = [kubernetes_namespace.external_secrets]
   }
   
   resource "kubernetes_namespace" "external_secrets" {
     metadata {
       name = "external-secrets-system"
       labels = {
         "name" = "external-secrets-system"
         "pod-security.kubernetes.io/enforce" = "restricted"
         "pod-security.kubernetes.io/audit"   = "restricted"
         "pod-security.kubernetes.io/warn"    = "restricted"
       }
     }
   }

2. Create SecretStore Configuration

   # File: modules/external-secrets/secret-store.tf
   resource "kubernetes_manifest" "scaleway_secret_store" {
     manifest = {
       apiVersion = "external-secrets.io/v1beta1"
       kind       = "SecretStore"
       metadata = {
         name      = "scaleway-secret-store"
         namespace = var.namespace
       }
       spec = {
         provider = {
           scaleway = {
             region    = var.scaleway_region
             projectId = var.scaleway_project_id
             accessKey = {
               secretRef = {
                 name = "scaleway-credentials"
                 key  = "access-key"
               }
             }
             secretKey = {
               secretRef = {
                 name = "scaleway-credentials"
                 key  = "secret-key"
               }
             }
           }
         }
       }
     }
   
     depends_on = [helm_release.external_secrets]
   }
   
   resource "kubernetes_secret" "scaleway_credentials" {
     metadata {
       name      = "scaleway-credentials"
       namespace = var.namespace
     }
   
     data = {
       access-key = var.scaleway_access_key
       secret-key = var.scaleway_secret_key
     }
   
     type = "Opaque"
   }

Day 5: Database Secret Integration

Owner: Database Team
Priority: P0

1. Create ExternalSecret for Database Credentials

   # File: modules/coder-deployment/external-secrets.tf
   resource "kubernetes_manifest" "database_external_secret" {
     manifest = {
       apiVersion = "external-secrets.io/v1beta1"
       kind       = "ExternalSecret"
       metadata = {
         name      = "database-credentials"
         namespace = "coder"
       }
       spec = {
         refreshInterval = "1h"
         secretStoreRef = {
           kind = "SecretStore"
           name = "scaleway-secret-store"
         }
         target = {
           name = "database-credentials"
           creationPolicy = "Owner"
           template = {
             type = "Opaque"
             data = {
               "POSTGRES_HOST"     = "{{ .host }}"
               "POSTGRES_PORT"     = "{{ .port }}"
               "POSTGRES_USER"     = "{{ .username }}"
               "POSTGRES_PASSWORD" = "{{ .password }}"
               "POSTGRES_DB"       = "{{ .database }}"
               "POSTGRES_SSLMODE"  = "{{ .ssl_mode }}"
               "DATABASE_URL"      = "postgres://{{ .username }}:{{ .password }}@{{ .host }}:{{ .port }}/{{ .database }}?sslmode={{ .ssl_mode }}"
             }
           }
         }
         data = [
           {
             secretKey = "host"
             remoteRef = {
               key      = "name:${var.environment}-database-credentials"
               property = "host"
             }
           },
           {
             secretKey = "port"
             remoteRef = {
               key      = "name:${var.environment}-database-credentials"
               property = "port"
             }
           },
           {
             secretKey = "username"
             remoteRef = {
               key      = "name:${var.environment}-database-credentials"
               property = "username"
             }
           },
           {
             secretKey = "password"
             remoteRef = {
               key      = "name:${var.environment}-database-credentials"
               property = "password"
             }
           },
           {
             secretKey = "database"
             remoteRef = {
               key      = "name:${var.environment}-database-credentials"
               property = "database"
             }
           },
           {
             secretKey = "ssl_mode"
             remoteRef = {
               key      = "name:${var.environment}-database-credentials"
               property = "ssl_mode"
             }
           }
         ]
       }
     }
   
     depends_on = [
       kubernetes_manifest.scaleway_secret_store,
       kubernetes_namespace.coder
     ]
   }

2. Update Coder Deployment to Use External Secrets

   # File: modules/coder-deployment/deployment.tf (update)
   resource "helm_release" "coder" {
     name       = "coder"
     repository = "https://helm.coder.com/v2"
     chart      = "coder"
     version    = var.coder_version
     namespace  = "coder"
   
     values = [
       yamlencode({
         coder = {
           env = [
             {
               name = "CODER_PG_CONNECTION_URL"
               valueFrom = {
                 secretKeyRef = {
                   name = "database-credentials"
                   key  = "DATABASE_URL"
                 }
               }
             },
             {
               name = "CODER_ACCESS_URL"
               value = "https://${var.coder_hostname}"
             }
           ]
         }
   
         security = {
           securityContext = {
             runAsNonRoot = true
             runAsUser    = 1000
             runAsGroup   = 1000
             fsGroup      = 1000
             allowPrivilegeEscalation = false
             readOnlyRootFilesystem   = true
             seccompProfile = {
               type = "RuntimeDefault"
             }
             capabilities = {
               drop = ["ALL"]
             }
           }
         }
       })
     ]
   
     depends_on = [
       kubernetes_manifest.database_external_secret
     ]
   }

Security Benefits

Immediate Security Improvements

1. Credential Protection

   • Database passwords no longer stored in Terraform state
   • API keys encrypted at rest with AES-256 KMS
   • Automatic credential rotation capabilities
   • Centralized audit logging of secret access

2. Access Control

   • Namespace-scoped RBAC for secret access
   • Service account isolation between components
   • Template-specific secret scoping
   • Time-based secret refresh (30m-1h intervals)

3. Compliance

   • SOC2 Type 2 compliant secret storage
   • Audit trail for all secret operations
   • Encryption in transit and at rest
   • Regional data residency compliance

Cost Analysis

Scaleway Secret Manager Pricing

• Storage: €0.40 per secret per month
• API Requests: €0.40 per 10K requests
• Estimated Monthly Cost: €15-25 per environment

Cost-Benefit Analysis

• Security Risk Reduction: High (prevents credential breaches)
• Operational Efficiency: Medium (reduced manual secret management)
• Compliance Value: High (meets enterprise security requirements)
• Total ROI: 300-500% within 6 months

Migration Plan

Pre-Migration Checklist

• Scaleway Secret Manager enabled in target regions
• IAM policies configured for External Secrets Operator
• External Secrets Operator tested in development
• Database connectivity validated with new secret injection
• Template secrets identified and catalogued
• RBAC policies reviewed and approved
• Rollback procedures documented and tested

Success Metrics

• Zero credentials exposed in Terraform state files
• 100% of secrets managed through Scaleway Secret Manager
• < 5 seconds secret injection latency in workspaces
• 99.9% secret availability SLA
• Zero security incidents related to credential management

Conclusion

Implementing Scaleway Secret Manager with External Secrets Operator addresses critical security vulnerabilities while providing enterprise-grade secret management capabilities. The solution eliminates credential exposure in Terraform state files, provides centralized secret lifecycle management, and enhances the overall security posture of the Coder infrastructure.

The phased implementation approach ensures minimal disruption while delivering immediate security benefits. The integration with Coder templates provides developers with seamless access to required secrets while maintaining strict security controls and audit capabilities.

---

Document Version: 1.0
Date: 2025-08-19
Authors: Security and Infrastructure Teams
Status: PROPOSAL - Ready for Implementation

[pacphi]

Implementation Plan by Use Cases V3 - Remote State Edition

Overview

This implementation plan addresses the security and optimization recommendations from the Terraform Security Review, organized by priority and structured as use cases for iterative development. Version 3 specifically accounts for the remote state management architecture using Scaleway Object Storage as the exclusive Terraform backend.

Remote State Architecture

This project uses a two-phase deployment with remote state stored exclusively in Scaleway Object Storage:

• Infrastructure Phase: State stored at {env}/infra/terraform.tfstate in bucket terraform-state-coder-{env}
• Coder Application Phase: State stored at {env}/coder/terraform.tfstate in bucket terraform-state-coder-{env}
• Cross-Phase Communication: Coder phase uses terraform_remote_state data source to access infrastructure outputs
• Script Access: All scripts use terraform init + terraform output pattern with proper backend authentication

Cross-Technology Integration Patterns

Each use case leverages capabilities across Terraform → Bash Scripts → GitHub Workflows with remote state as the central coordination mechanism:

1. Terraform: Creates infrastructure, stores outputs in remote state
2. Bash Scripts: Access Terraform outputs via terraform output commands after backend initialization
3. GitHub Workflows: Coordinate deployment phases, manage backend configurations, consume outputs via artifacts

---

Priority 0 (P0) - Critical Security Issues

UC-P0-001: Input Validation and Command Injection Prevention

User Story: As a platform administrator, I want comprehensive input validation to prevent command injection attacks through user-provided variables.

Current State: Basic validation exists but lacks comprehensive sanitization for all user inputs including environment names, template names, and configuration values.

Remote State Considerations:

• Backend bucket names derived from environment names must be validated
• State key paths constructed from user inputs require sanitization
• Cross-phase data access validation needed

Acceptance Criteria:

• All user inputs are validated using allowlists for known safe values
• Command injection attempts are blocked and logged
• Remote state bucket names and keys are validated before use
• Cross-phase data integrity checks implemented

Cross-Technology Implementation:

Terraform Foundations (modules/validation/):

# modules/validation/main.tf
variable "environment_name" {
  description = "Environment name with validation"
  type        = string
  validation {
    condition = can(regex("^(dev|staging|prod)$", var.environment_name))
    error_message = "Environment must be dev, staging, or prod."
  }
}

variable "template_name" {
  description = "Template name with validation"
  type        = string
  validation {
    condition = can(regex("^[a-z0-9-]+$", var.template_name))
    error_message = "Template name must contain only lowercase letters, numbers, and hyphens."
  }
}

# Remote state bucket validation
locals {
  validated_bucket_name = "terraform-state-coder-${var.environment_name}"
  validated_state_key   = "${var.environment_name}/${var.phase}/terraform.tfstate"
}

Bash Script Integration (scripts/lib/universal-validation.sh):

#!/bin/bash
# Universal input validation library with remote state support

validate_environment() {
    local env="$1"
    if [[ ! "$env" =~ ^(dev|staging|prod)$ ]]; then
        log_error "Invalid environment: $env. Must be dev, staging, or prod."
        return 1
    fi
}

validate_terraform_outputs() {
    local env="$1"
    local phase="$2"
    local env_dir="environments/${env}/${phase}"

    cd "$env_dir"

    # Initialize backend safely
    if ! terraform init -input=false; then
        log_error "Failed to initialize Terraform backend for ${env}/${phase}"
        return 1
    fi

    # Validate remote state accessibility
    if ! terraform show -json >/dev/null 2>&1; then
        log_error "Cannot access remote state for ${env}/${phase}"
        return 1
    fi

    # Extract and validate required outputs
    local required_outputs=("cluster_endpoint" "database_host" "access_url")
    for output in "${required_outputs[@]}"; do
        if ! terraform output "$output" >/dev/null 2>&1; then
            log_error "Required output missing from remote state: $output"
            return 1
        fi
    done
}

# Sanitize inputs for shell command construction
sanitize_shell_input() {
    local input="$1"
    # Remove dangerous characters
    echo "$input" | sed 's/[;&|`$(){}[\]*?~<>^!]//' | tr -d '\n\r'
}

GitHub Workflow Integration (.github/workflows/validate-inputs.yml):

name: Validate All Inputs

on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string
      template:
        required: false
        type: string

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - name: Validate Environment Input
        run: |
          if [[ ! "${{ inputs.environment }}" =~ ^(dev|staging|prod)$ ]]; then
            echo "❌ Invalid environment: ${{ inputs.environment }}"
            exit 1
          fi

      - name: Validate Template Input
        if: inputs.template != ''
        run: |
          if [[ ! "${{ inputs.template }}" =~ ^[a-z0-9-]+$ ]]; then
            echo "❌ Invalid template name: ${{ inputs.template }}"
            exit 1
          fi

      - name: Validate Remote State Access
        run: |
          # Download backend configuration from previous job
          # Initialize and verify remote state connectivity
          cd environments/${{ inputs.environment }}/infra
          terraform init -input=false
          terraform show -json > /dev/null

Implementation Steps:

1. Create universal validation module in Terraform
2. Implement bash validation library with remote state checks
3. Add workflow input validation with backend verification
4. Update all existing scripts to use validation functions
5. Add logging and monitoring for validation failures

Validation Procedures:

• Test with malicious inputs including command injection attempts
• Verify remote state access validation works across all phases
• Confirm cross-technology validation consistency

---

UC-P0-002: Scaleway Secret Manager Integration with External Secrets Operator

User Story: As a security engineer, I want to migrate all hardcoded secrets to Scaleway Secret Manager and automatically inject them into Kubernetes using External Secrets Operator.

Current State: Database credentials and other secrets are stored as Kubernetes secrets or environment variables without proper secret management.

Remote State Integration:

• Infrastructure phase creates Secret Manager resources, stores ARNs in remote state
• Coder phase accesses secret ARNs from infrastructure remote state
• Scripts retrieve secret metadata through Terraform outputs

Acceptance Criteria:

• All secrets stored in Scaleway Secret Manager with AES-256 KMS encryption
• External Secrets Operator automatically syncs secrets to Kubernetes
• No hardcoded secrets in code or remote state
• Secret rotation capabilities implemented

Cross-Technology Implementation:

Terraform Infrastructure Phase (modules/scaleway-secrets/):

# modules/scaleway-secrets/main.tf
resource "scaleway_secret" "database_credentials" {
  name        = "coder-db-credentials-${var.environment}"
  description = "Database credentials for Coder ${var.environment}"

  tags = merge(var.tags, {
    Purpose = "database-access"
    ManagedBy = "terraform"
  })
}

resource "scaleway_secret_version" "database_credentials" {
  secret_id = scaleway_secret.database_credentials.id
  data = jsonencode({
    username = var.db_username
    password = var.db_password
    host     = var.db_host
    port     = var.db_port
    database = var.db_name
  })
}

# Store secret ARN in outputs for remote state access
output "database_secret_arn" {
  description = "ARN of database credentials secret"
  value       = scaleway_secret.database_credentials.arn
  sensitive   = false  # ARN is not sensitive, actual secret data is
}

output "secret_manager_config" {
  description = "Secret Manager configuration for External Secrets Operator"
  value = {
    region     = var.scaleway_region
    secret_arns = {
      database = scaleway_secret.database_credentials.arn
      coder_auth = scaleway_secret.coder_auth_credentials.arn
    }
  }
  sensitive = false
}

Terraform Coder Phase (environments/{env}/coder/main.tf):

# Access infrastructure outputs from remote state
data "terraform_remote_state" "infra" {
  backend = "s3"
  config = {
    bucket = "terraform-state-coder-${var.environment}"
    key    = "${var.environment}/infra/terraform.tfstate"
    region = var.scaleway_region
    endpoints = {
      s3 = "https://s3.${var.scaleway_region}.scw.cloud"
    }
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
    skip_requesting_account_id  = true
  }
}

# External Secrets Operator deployment
resource "helm_release" "external_secrets" {
  name       = "external-secrets"
  namespace  = "external-secrets-system"
  repository = "https://charts.external-secrets.io"
  chart      = "external-secrets"
  version    = "0.9.11"

  create_namespace = true

  values = [yamlencode({
    installCRDs = true
    secretStore = {
      scaleway = {
        region = var.scaleway_region
      }
    }
  })]
}

# ClusterSecretStore for Scaleway Secret Manager
resource "kubectl_manifest" "scaleway_cluster_secret_store" {
  yaml_body = yamlencode({
    apiVersion = "external-secrets.io/v1beta1"
    kind       = "ClusterSecretStore"
    metadata = {
      name = "scaleway-secret-store"
    }
    spec = {
      provider = {
        scaleway = {
          region = var.scaleway_region
          auth = {
            accessKey = {
              secretRef = {
                name = "scaleway-credentials"
                key  = "access-key"
              }
            }
            secretKey = {
              secretRef = {
                name = "scaleway-credentials"
                key  = "secret-key"
              }
            }
          }
        }
      }
    }
  })
}

# ExternalSecret for database credentials using remote state ARN
resource "kubectl_manifest" "database_external_secret" {
  yaml_body = yamlencode({
    apiVersion = "external-secrets.io/v1beta1"
    kind       = "ExternalSecret"
    metadata = {
      name      = "coder-database-secret"
      namespace = "coder"
    }
    spec = {
      refreshInterval = "1h"
      secretStoreRef = {
        name = "scaleway-secret-store"
        kind = "ClusterSecretStore"
      }
      target = {
        name = "coder-database-credentials"
        creationPolicy = "Owner"
      }
      data = [
        {
          secretKey = "DB_URL"
          remoteRef = {
            key = data.terraform_remote_state.infra.outputs.secret_manager_config.secret_arns.database
            property = "database_url"
          }
        }
      ]
    }
  })
}

Bash Script Integration (scripts/lib/secrets-manager.sh):

#!/bin/bash
# Secrets Manager integration library with remote state support

get_secret_arn_from_state() {
    local env="$1"
    local secret_name="$2"
    local phase="${3:-infra}"  # Default to infrastructure phase

    local env_dir="environments/${env}/${phase}"
    cd "$env_dir"

    # Initialize backend
    terraform init -input=false >/dev/null 2>&1

    # Get secret ARN from remote state outputs
    local secret_arn
    secret_arn=$(terraform output -json secret_manager_config 2>/dev/null | \
                jq -r ".secret_arns.${secret_name} // empty")

    if [[ -z "$secret_arn" ]]; then
        log_error "Secret ARN not found in remote state: $secret_name"
        return 1
    fi

    echo "$secret_arn"
}

rotate_database_secret() {
    local env="$1"

    log "Starting database secret rotation for environment: $env"

    # Get current secret ARN from infrastructure remote state
    local secret_arn
    secret_arn=$(get_secret_arn_from_state "$env" "database" "infra")

    if [[ -z "$secret_arn" ]]; then
        log_error "Failed to get database secret ARN from remote state"
        return 1
    fi

    # Generate new password
    local new_password
    new_password=$(openssl rand -base64 32)

    # Update secret in Scaleway Secret Manager
    scw secret version create \
        secret-id="$secret_arn" \
        data="{\"password\":\"$new_password\"}" \
        --wait

    # Trigger External Secrets Operator refresh
    kubectl annotate externalsecret coder-database-secret \
        force-sync="$(date +%s)" -n coder

    log_success "Database secret rotation completed"
}

validate_secrets_in_kubernetes() {
    local env="$1"

    # Check if External Secrets Operator is running
    if ! kubectl get pods -n external-secrets-system | grep -q "Running"; then
        log_error "External Secrets Operator not running"
        return 1
    fi

    # Validate secret sync status
    local secret_status
    secret_status=$(kubectl get externalsecret coder-database-secret -n coder -o jsonpath='{.status.conditions[0].status}')

    if [[ "$secret_status" != "True" ]]; then
        log_error "External secret sync failed"
        kubectl describe externalsecret coder-database-secret -n coder
        return 1
    fi

    log_success "All secrets successfully synced from Scaleway Secret Manager"
}

GitHub Workflow Integration (.github/workflows/manage-secrets.yml):

name: Manage Secrets

on:
  workflow_dispatch:
    inputs:
      environment:
        required: true
        type: choice
        options: [dev, staging, prod]
      action:
        required: true
        type: choice
        options: [rotate, validate, setup]

jobs:
  manage-secrets:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v5

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "~1.12.0"

      - name: Download Backend Configuration
        uses: actions/download-artifact@v5
        with:
          name: backend-config-${{ inputs.environment }}-infra-${{ github.run_id }}
          path: environments/${{ inputs.environment }}/infra

      - name: Execute Secret Management Action
        env:
          SCW_ACCESS_KEY: ${{ secrets.SCW_ACCESS_KEY }}
          SCW_SECRET_KEY: ${{ secrets.SCW_SECRET_KEY }}
        run: |
          source scripts/lib/secrets-manager.sh

          case "${{ inputs.action }}" in
            rotate)
              rotate_database_secret "${{ inputs.environment }}"
              ;;
            validate)
              validate_secrets_in_kubernetes "${{ inputs.environment }}"
              ;;
            setup)
              # Setup External Secrets Operator if not exists
              scripts/secrets/setup-external-secrets.sh --env="${{ inputs.environment }}"
              ;;
          esac

Implementation Steps:

1. Deploy Scaleway Secret Manager module in infrastructure phase
2. Update Coder deployment to use External Secrets Operator
3. Create bash library for secret management with remote state integration
4. Add GitHub workflows for secret rotation and validation
5. Migrate existing hardcoded secrets

Validation Procedures:

• Test secret creation and rotation across all environments
• Verify External Secrets Operator sync functionality
• Confirm no secrets are accessible in remote state files

---

UC-P0-003: Pod Security Standards Implementation

User Story: As a security engineer, I want to enforce strict Pod Security Standards across all environments to prevent privilege escalation and container breakouts.

Current State: Basic security policies exist but lack comprehensive Pod Security Standards enforcement and environment-specific configuration.

Remote State Integration:

• Infrastructure phase deploys security policies, stores configuration in remote state
• Coder phase references security settings from infrastructure remote state
• Scripts validate security compliance using remote state data

Acceptance Criteria:

• Pod Security Standards "restricted" enforced in production
• Environment-specific security policies (baseline for dev, restricted for prod)
• All workspaces comply with security policies
• Security violations blocked and logged

Cross-Technology Implementation:

Terraform Infrastructure Phase (modules/security/pod-security.tf):

# Environment-specific Pod Security Standards
locals {
  pod_security_config = {
    dev = {
      enforce = "baseline"
      audit   = "restricted"
      warn    = "restricted"
    }
    staging = {
      enforce = "restricted"
      audit   = "restricted"
      warn    = "restricted"
    }
    prod = {
      enforce = "restricted"
      audit   = "restricted"
      warn    = "restricted"
    }
  }

  security_config = local.pod_security_config[var.environment]
}

# Apply Pod Security Standards to namespaces
resource "kubectl_manifest" "pod_security_policy" {
  for_each = toset(["coder", "coder-workspaces", "external-secrets-system"])

  yaml_body = yamlencode({
    apiVersion = "v1"
    kind       = "Namespace"
    metadata = {
      name = each.value
      labels = merge(
        {
          "pod-security.kubernetes.io/enforce" = local.security_config.enforce
          "pod-security.kubernetes.io/audit"   = local.security_config.audit
          "pod-security.kubernetes.io/warn"    = local.security_config.warn
        },
        var.environment == "prod" ? {
          "pod-security.kubernetes.io/enforce-version" = "latest"
        } : {}
      )
    }
  })
}

# Security context constraints for Coder workspaces
resource "kubectl_manifest" "workspace_security_context" {
  yaml_body = yamlencode({
    apiVersion = "v1"
    kind       = "SecurityContextConstraints"
    metadata = {
      name = "coder-workspace-scc"
    }
    allowHostDirVolumePlugin = false
    allowHostIPC            = false
    allowHostNetwork        = false
    allowHostPID            = false
    allowHostPorts          = false
    allowPrivilegedContainer = false
    allowedCapabilities     = null
    defaultAddCapabilities  = null
    requiredDropCapabilities = ["ALL"]
    fsGroup = {
      type = "MustRunAs"
      ranges = [
        {
          min = 1000
          max = 65534
        }
      ]
    }
    runAsUser = {
      type = "MustRunAsNonRoot"
    }
    seLinuxContext = {
      type = "MustRunAs"
    }
    volumes = [
      "configMap",
      "downwardAPI",
      "emptyDir",
      "persistentVolumeClaim",
      "projected",
      "secret"
    ]
  })
}

# Store security configuration in remote state for Coder phase
output "pod_security_config" {
  description = "Pod Security Standards configuration"
  value = {
    enforce_level = local.security_config.enforce
    audit_level   = local.security_config.audit
    warn_level    = local.security_config.warn
    namespaces    = ["coder", "coder-workspaces", "external-secrets-system"]
  }
}

output "security_context_constraints" {
  description = "Security context constraints for workspaces"
  value = {
    allowed_volumes = ["configMap", "downwardAPI", "emptyDir", "persistentVolumeClaim", "projected", "secret"]
    run_as_non_root = true
    drop_capabilities = ["ALL"]
  }
}

Terraform Coder Phase (modules/coder-deployment/security.tf):

# Access security configuration from infrastructure remote state
data "terraform_remote_state" "infra" {
  # ... backend configuration
}

locals {
  security_config = data.terraform_remote_state.infra.outputs.pod_security_config
  security_constraints = data.terraform_remote_state.infra.outputs.security_context_constraints
}

# Coder deployment with security context
resource "kubernetes_deployment" "coder" {
  metadata {
    name      = "coder"
    namespace = "coder"
  }

  spec {
    template {
      spec {
        security_context {
          run_as_non_root = true
          run_as_user     = 1000
          fs_group        = 1000
          supplemental_groups = [1000]
        }

        container {
          name  = "coder"
          image = "ghcr.io/coder/coder:${var.coder_version}"

          security_context {
            allow_privilege_escalation = false
            capabilities {
              drop = ["ALL"]
            }
            read_only_root_filesystem = true
            run_as_non_root          = true
            run_as_user              = 1000
            seccomp_profile {
              type = "RuntimeDefault"
            }
          }

          # Apply security constraints from remote state
          resources {
            limits = {
              cpu    = "1000m"
              memory = "2Gi"
            }
            requests = {
              cpu    = "100m"
              memory = "256Mi"
            }
          }
        }
      }
    }
  }
}

# Template security policy for workspaces
resource "coder_template" "secure_workspace" {
  count = var.enable_secure_templates ? 1 : 0

  name         = "secure-development"
  description  = "Security-hardened development workspace"
  default_ttl  = "8h"
  max_ttl      = "24h"

  # Embed security constraints from infrastructure remote state
  template = templatefile("${path.module}/templates/secure-workspace.tf.tpl", {
    security_constraints = local.security_constraints
    pod_security_level  = local.security_config.enforce_level
    allowed_volumes     = jsonencode(local.security_constraints.allowed_volumes)
  })
}

Bash Script Integration (scripts/security/validate-pod-security.sh):

#!/bin/bash
# Pod Security Standards validation with remote state integration

validate_pod_security_compliance() {
    local env="$1"

    log "Validating Pod Security Standards compliance for environment: $env"

    # Get security configuration from infrastructure remote state
    cd "environments/${env}/infra"
    terraform init -input=false >/dev/null 2>&1

    local security_config
    security_config=$(terraform output -json pod_security_config 2>/dev/null)

    if [[ -z "$security_config" ]]; then
        log_error "Cannot retrieve security configuration from remote state"
        return 1
    fi

    local enforce_level
    enforce_level=$(echo "$security_config" | jq -r '.enforce_level')

    local namespaces
    namespaces=$(echo "$security_config" | jq -r '.namespaces[]')

    # Validate namespace labels
    while IFS= read -r namespace; do
        local current_enforce
        current_enforce=$(kubectl get namespace "$namespace" \
            -o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}' 2>/dev/null || echo "")

        if [[ "$current_enforce" != "$enforce_level" ]]; then
            log_error "Namespace $namespace has incorrect Pod Security Standard: $current_enforce (expected: $enforce_level)"
            return 1
        fi

        log_success "Namespace $namespace correctly enforces Pod Security Standard: $enforce_level"
    done <<< "$namespaces"

    # Validate running pods compliance
    validate_running_pods_compliance "$env"
}

validate_running_pods_compliance() {
    local env="$1"

    log "Checking running pods for security compliance..."

    # Check for pods running as root
    local root_pods
    root_pods=$(kubectl get pods --all-namespaces -o json | \
        jq -r '.items[] | select(.spec.securityContext.runAsUser == 0 or .spec.containers[].securityContext.runAsUser == 0) | "\(.metadata.namespace)/\(.metadata.name)"')

    if [[ -n "$root_pods" ]]; then
        log_error "Found pods running as root:"
        echo "$root_pods" | while read -r pod; do
            log_error "  - $pod"
        done
        return 1
    fi

    # Check for privileged containers
    local privileged_pods
    privileged_pods=$(kubectl get pods --all-namespaces -o json | \
        jq -r '.items[] | select(.spec.containers[].securityContext.privileged == true) | "\(.metadata.namespace)/\(.metadata.name)"')

    if [[ -n "$privileged_pods" ]]; then
        log_error "Found privileged containers:"
        echo "$privileged_pods" | while read -r pod; do
            log_error "  - $pod"
        done
        return 1
    fi

    log_success "All running pods comply with security standards"
}

audit_security_violations() {
    local env="$1"
    local days="${2:-7}"

    log "Auditing security violations for the last $days days..."

    # Check audit logs for Pod Security Standards violations
    kubectl get events --all-namespaces \
        --field-selector reason=FailedCreate \
        --output json | \
        jq -r --arg days "$days" '
            .items[] |
            select(.lastTimestamp | fromdateiso8601 > (now - ($days | tonumber * 86400))) |
            select(.message | contains("violates PodSecurity")) |
            "\(.lastTimestamp) \(.namespace)/\(.involvedObject.name): \(.message)"
        ' | while read -r violation; do
        log_warning "Security violation: $violation"
    done
}

GitHub Workflow Integration (.github/workflows/security-compliance.yml):

name: Security Compliance Check

on:
  schedule:
    - cron: '0 6 * * *'  # Daily at 6 AM
  workflow_dispatch:
    inputs:
      environment:
        required: true
        type: choice
        options: [dev, staging, prod]

jobs:
  security-compliance:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        environment: ${{ github.event_name == 'workflow_dispatch' && fromJSON(format('["{0}"]', inputs.environment)) || fromJSON('["dev", "staging", "prod"]') }}

    steps:
      - name: Checkout
        uses: actions/checkout@v5

      - name: Setup Tools
        uses: azure/setup-kubectl@v4
        with:
          version: 'v1.32.0'

      - name: Download Backend Configuration
        uses: actions/download-artifact@v5
        with:
          name: backend-config-${{ matrix.environment }}-infra-${{ github.run_id }}
          path: environments/${{ matrix.environment }}/infra

      - name: Validate Pod Security Compliance
        env:
          SCW_ACCESS_KEY: ${{ secrets.SCW_ACCESS_KEY }}
          SCW_SECRET_KEY: ${{ secrets.SCW_SECRET_KEY }}
        run: |
          source scripts/security/validate-pod-security.sh
          validate_pod_security_compliance "${{ matrix.environment }}"

      - name: Audit Security Violations
        run: |
          source scripts/security/validate-pod-security.sh
          audit_security_violations "${{ matrix.environment }}" 7

      - name: Create Security Report
        if: failure()
        run: |
          mkdir -p security-reports
          cat > security-reports/pod-security-${{ matrix.environment }}.md << EOF
          # Pod Security Compliance Report

          **Environment**: ${{ matrix.environment }}
          **Date**: $(date)
          **Status**: Failed

          ## Issues Found

          - Check workflow logs for specific violations
          - Review Pod Security Standards configuration
          - Audit running workloads for compliance

          ## Remediation

          1. Update non-compliant workloads
          2. Review security constraints in remote state
          3. Re-run compliance validation
          EOF

      - name: Upload Security Report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: security-report-${{ matrix.environment }}
          path: security-reports/

Implementation Steps:

1. Deploy Pod Security Standards module in infrastructure phase
2. Update Coder deployment templates with security contexts
3. Create validation scripts with remote state integration
4. Set up automated compliance checking workflows
5. Configure monitoring and alerting for violations

Validation Procedures:

• Test Pod Security Standards enforcement across environments
• Verify workspace templates comply with security policies
• Confirm security violations are properly blocked and logged

---

UC-P0-004: RBAC Least Privilege Implementation

User Story: As a security administrator, I want to implement least privilege RBAC policies across all Kubernetes namespaces to minimize attack surface.

Current State: Basic RBAC exists but lacks fine-grained permissions and environment-specific policies.

Remote State Integration:

• Infrastructure phase creates RBAC policies, stores service account details in remote state
• Coder phase accesses RBAC configuration from infrastructure remote state
• Scripts validate RBAC compliance using remote state metadata

Acceptance Criteria:

• All services use dedicated service accounts with minimal permissions
• Environment-specific RBAC policies (dev has broader access than prod)
• Regular RBAC auditing and unused permission cleanup
• Cross-namespace access properly controlled

Cross-Technology Implementation:

Terraform Infrastructure Phase (modules/rbac/main.tf):

# Environment-specific RBAC configuration
locals {
  rbac_config = {
    dev = {
      coder_permissions = ["get", "list", "watch", "create", "update", "patch", "delete"]
      workspace_permissions = ["get", "list", "watch", "create", "update", "patch", "delete"]
      debug_access = true
    }
    staging = {
      coder_permissions = ["get", "list", "watch", "create", "update", "patch"]
      workspace_permissions = ["get", "list", "watch", "create", "update", "patch"]
      debug_access = true
    }
    prod = {
      coder_permissions = ["get", "list", "watch", "create", "update"]
      workspace_permissions = ["get", "list", "watch", "create"]
      debug_access = false
    }
  }

  environment_rbac = local.rbac_config[var.environment]
}

# Coder service account
resource "kubernetes_service_account" "coder" {
  metadata {
    name      = "coder"
    namespace = "coder"
    labels = {
      "app.kubernetes.io/name"     = "coder"
      "app.kubernetes.io/instance" = var.environment
      "managed-by"                 = "terraform"
    }
  }

  automount_service_account_token = true
}

# Coder ClusterRole with environment-specific permissions
resource "kubernetes_cluster_role" "coder" {
  metadata {
    name = "coder-${var.environment}"
    labels = {
      "app.kubernetes.io/name" = "coder"
      "environment"            = var.environment
    }
  }

  # Core Coder permissions
  rule {
    api_groups = [""]
    resources  = ["pods", "pods/log", "pods/exec"]
    verbs      = local.environment_rbac.coder_permissions
  }

  rule {
    api_groups = [""]
    resources  = ["services", "persistentvolumeclaims", "secrets", "configmaps"]
    verbs      = local.environment_rbac.coder_permissions
  }

  rule {
    api_groups = ["apps"]
    resources  = ["deployments", "replicasets"]
    verbs      = local.environment_rbac.coder_permissions
  }

  # Workspace management permissions
  rule {
    api_groups = [""]
    resources  = ["namespaces"]
    verbs      = local.environment_rbac.workspace_permissions
    resource_names = ["coder-workspaces-*"]
  }

  # Environment-specific debug access
  dynamic "rule" {
    for_each = local.environment_rbac.debug_access ? [1] : []
    content {
      api_groups = [""]
      resources  = ["pods/portforward", "pods/proxy"]
      verbs      = ["create", "get"]
    }
  }

  # Read-only access to cluster resources
  rule {
    api_groups = [""]
    resources  = ["nodes", "namespaces"]
    verbs      = ["get", "list", "watch"]
  }
}

# Bind Coder service account to ClusterRole
resource "kubernetes_cluster_role_binding" "coder" {
  metadata {
    name = "coder-${var.environment}"
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = kubernetes_cluster_role.coder.metadata[0].name
  }

  subject {
    kind      = "ServiceAccount"
    name      = kubernetes_service_account.coder.metadata[0].name
    namespace = kubernetes_service_account.coder.metadata[0].namespace
  }
}

# Workspace template service account
resource "kubernetes_service_account" "workspace_template" {
  metadata {
    name      = "workspace-template"
    namespace = "coder"
  }

  automount_service_account_token = false
}

# Minimal permissions for workspace templates
resource "kubernetes_role" "workspace_template" {
  metadata {
    namespace = "coder"
    name      = "workspace-template"
  }

  rule {
    api_groups = [""]
    resources  = ["configmaps"]
    verbs      = ["get", "list"]
    resource_names = ["workspace-*"]
  }
}

resource "kubernetes_role_binding" "workspace_template" {
  metadata {
    name      = "workspace-template"
    namespace = "coder"
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "Role"
    name      = kubernetes_role.workspace_template.metadata[0].name
  }

  subject {
    kind      = "ServiceAccount"
    name      = kubernetes_service_account.workspace_template.metadata[0].name
    namespace = kubernetes_service_account.workspace_template.metadata[0].namespace
  }
}

# Output RBAC configuration for remote state access
output "rbac_config" {
  description = "RBAC configuration for Coder deployment"
  value = {
    coder_service_account = {
      name      = kubernetes_service_account.coder.metadata[0].name
      namespace = kubernetes_service_account.coder.metadata[0].namespace
    }
    workspace_service_account = {
      name      = kubernetes_service_account.workspace_template.metadata[0].name
      namespace = kubernetes_service_account.workspace_template.metadata[0].namespace
    }
    cluster_role = {
      name = kubernetes_cluster_role.coder.metadata[0].name
    }
    permissions = local.environment_rbac
  }
}

output "rbac_audit_config" {
  description = "RBAC audit configuration"
  value = {
    service_accounts = [
      "${kubernetes_service_account.coder.metadata[0].namespace}:${kubernetes_service_account.coder.metadata[0].name}",
      "${kubernetes_service_account.workspace_template.metadata[0].namespace}:${kubernetes_service_account.workspace_template.metadata[0].name}"
    ]
    cluster_roles = [kubernetes_cluster_role.coder.metadata[0].name]
    environment = var.environment
  }
}

Terraform Coder Phase (environments/{env}/coder/main.tf):

# Access RBAC configuration from infrastructure remote state
data "terraform_remote_state" "infra" {
  # ... backend configuration
}

locals {
  rbac_config = data.terraform_remote_state.infra.outputs.rbac_config
}

# Coder deployment using RBAC from remote state
module "coder_deployment" {
  source = "../../../modules/coder-deployment"

  # Use service account from infrastructure phase
  service_account_name = local.rbac_config.coder_service_account.name

  # Apply RBAC constraints to Coder configuration
  rbac_enabled = true
  rbac_config  = local.rbac_config.permissions

  # Other configuration...
}

# Additional workspace-specific RBAC
resource "kubernetes_role" "workspace_user" {
  for_each = var.workspace_namespaces

  metadata {
    namespace = each.value
    name      = "workspace-user"
  }

  # Minimal permissions for workspace users
  rule {
    api_groups = [""]
    resources  = ["pods", "pods/log"]
    verbs      = ["get", "list", "watch"]
  }

  rule {
    api_groups = [""]
    resources  = ["pods/exec", "pods/portforward"]
    verbs      = local.rbac_config.permissions.debug_access ? ["create"] : []
  }
}

Bash Script Integration (scripts/security/rbac-audit.sh):

#!/bin/bash
# RBAC audit and validation with remote state integration

audit_rbac_permissions() {
    local env="$1"

    log "Auditing RBAC permissions for environment: $env"

    # Get RBAC configuration from infrastructure remote state
    cd "environments/${env}/infra"
    terraform init -input=false >/dev/null 2>&1

    local rbac_config
    rbac_config=$(terraform output -json rbac_audit_config 2>/dev/null)

    if [[ -z "$rbac_config" ]]; then
        log_error "Cannot retrieve RBAC configuration from remote state"
        return 1
    fi

    # Extract configuration
    local service_accounts
    service_accounts=$(echo "$rbac_config" | jq -r '.service_accounts[]')

    local cluster_roles
    cluster_roles=$(echo "$rbac_config" | jq -r '.cluster_roles[]')

    # Audit service accounts
    log "Auditing service accounts..."
    while IFS= read -r sa; do
        local namespace="${sa%:*}"
        local name="${sa#*:}"

        if kubectl get serviceaccount "$name" -n "$namespace" >/dev/null 2>&1; then
            log_success "Service account exists: $namespace/$name"

            # Check for unnecessary tokens
            local token_secret
            token_secret=$(kubectl get serviceaccount "$name" -n "$namespace" \
                -o jsonpath='{.secrets[0].name}' 2>/dev/null || echo "")

            if [[ -n "$token_secret" ]]; then
                log_warning "Service account $namespace/$name has legacy token secret: $token_secret"
            fi
        else
            log_error "Service account missing: $namespace/$name"
        fi
    done <<< "$service_accounts"

    # Audit cluster roles
    log "Auditing cluster roles..."
    while IFS= read -r role; do
        if kubectl get clusterrole "$role" >/dev/null 2>&1; then
            log_success "Cluster role exists: $role"

            # Check for overly broad permissions
            audit_cluster_role_permissions "$role" "$env"
        else
            log_error "Cluster role missing: $role"
        fi
    done <<< "$cluster_roles"
}

audit_cluster_role_permissions() {
    local role="$1"
    local env="$2"

    # Get role permissions
    local permissions
    permissions=$(kubectl get clusterrole "$role" -o json)

    # Check for wildcard permissions
    local wildcard_rules
    wildcard_rules=$(echo "$permissions" | jq -r '.rules[] | select(.resources[] == "*" or .verbs[] == "*")')

    if [[ -n "$wildcard_rules" ]]; then
        log_warning "Cluster role $role has wildcard permissions (review for production)"
        if [[ "$env" == "prod" ]]; then
            log_error "Wildcard permissions not allowed in production environment"
            return 1
        fi
    fi

    # Check for dangerous permissions
    local dangerous_verbs=("delete" "deletecollection" "create" "update" "patch")
    for verb in "${dangerous_verbs[@]}"; do
        local dangerous_rules
        dangerous_rules=$(echo "$permissions" | jq -r --arg verb "$verb" '.rules[] | select(.verbs[] == $verb)')

        if [[ -n "$dangerous_rules" && "$env" == "prod" ]]; then
            log_warning "Cluster role $role has $verb permissions in production (ensure justified)"
        fi
    done
}

cleanup_unused_rbac() {
    local env="$1"

    log "Cleaning up unused RBAC resources for environment: $env"

    # Find unused service accounts
    local all_sa
    all_sa=$(kubectl get serviceaccounts --all-namespaces -o json | \
        jq -r '.items[] | "\(.metadata.namespace):\(.metadata.name)"')

    while IFS= read -r sa; do
        local namespace="${sa%:*}"
        local name="${sa#*:}"

        # Skip default and system service accounts
        if [[ "$name" == "default" || "$namespace" =~ ^kube- ]]; then
            continue
        fi

        # Check if service account is referenced by pods
        local pod_count
        pod_count=$(kubectl get pods -n "$namespace" \
            -o jsonpath='{.items[?(@.spec.serviceAccountName=="'$name'")].metadata.name}' | \
            wc -w)

        if [[ "$pod_count" -eq 0 ]]; then
            log_warning "Unused service account found: $namespace/$name"

            # In dry run mode, don't actually delete
            if [[ "${DRY_RUN:-false}" != "true" ]]; then
                read -p "Delete unused service account $namespace/$name? [y/N]: " -r
                if [[ $REPLY =~ ^[Yy]$ ]]; then
                    kubectl delete serviceaccount "$name" -n "$namespace"
                    log_success "Deleted unused service account: $namespace/$name"
                fi
            fi
        fi
    done <<< "$all_sa"
}

validate_rbac_least_privilege() {
    local env="$1"

    log "Validating RBAC least privilege compliance for environment: $env"

    # Get expected permissions from remote state
    cd "environments/${env}/infra"
    local rbac_config
    rbac_config=$(terraform output -json rbac_config 2>/dev/null)

    local expected_permissions
    expected_permissions=$(echo "$rbac_config" | jq -r '.permissions')

    # Validate Coder service account permissions match expected
    local coder_sa_name
    coder_sa_name=$(echo "$rbac_config" | jq -r '.coder_service_account.name')

    local coder_sa_namespace
    coder_sa_namespace=$(echo "$rbac_config" | jq -r '.coder_service_account.namespace')

    # Check cluster role binding
    local cluster_role_binding
    cluster_role_binding=$(kubectl get clusterrolebinding -o json | \
        jq -r --arg sa "$coder_sa_name" --arg ns "$coder_sa_namespace" \
        '.items[] | select(.subjects[]? | select(.kind == "ServiceAccount" and .name == $sa and .namespace == $ns))')

    if [[ -z "$cluster_role_binding" ]]; then
        log_error "Coder service account $coder_sa_namespace/$coder_sa_name is not bound to any cluster role"
        return 1
    fi

    log_success "RBAC configuration validated for environment: $env"
}

GitHub Workflow Integration (.github/workflows/rbac-management.yml):

name: RBAC Management

on:
  schedule:
    - cron: '0 3 * * 1'  # Weekly on Monday at 3 AM
  workflow_dispatch:
    inputs:
      environment:
        required: true
        type: choice
        options: [dev, staging, prod]
      action:
        required: true
        type: choice
        options: [audit, cleanup, validate]

jobs:
  rbac-management:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        environment: ${{ github.event_name == 'workflow_dispatch' && fromJSON(format('["{0}"]', inputs.environment)) || fromJSON('["dev", "staging", "prod"]') }}

    steps:
      - name: Checkout
        uses: actions/checkout@v5

      - name: Setup Tools
        uses: azure/setup-kubectl@v4
        with:
          version: 'v1.32.0'

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "~1.12.0"
          terraform_wrapper: false

      - name: Download Backend Configuration
        uses: actions/download-artifact@v5
        with:
          name: backend-config-${{ matrix.environment }}-infra-${{ github.run_id }}
          path: environments/${{ matrix.environment }}/infra

      - name: Download Kubeconfig
        uses: actions/download-artifact@v5
        with:
          name: kubeconfig-${{ matrix.environment }}-infra
          path: .

      - name: Setup Kubernetes Access
        run: |
          mkdir -p ~/.kube
          cp kubeconfig-${{ matrix.environment }} ~/.kube/config
          chmod 600 ~/.kube/config

      - name: Execute RBAC Management Action
        env:
          SCW_ACCESS_KEY: ${{ secrets.SCW_ACCESS_KEY }}
          SCW_SECRET_KEY: ${{ secrets.SCW_SECRET_KEY }}
          DRY_RUN: ${{ github.event_name == 'schedule' && 'true' || 'false' }}
        run: |
          source scripts/security/rbac-audit.sh

          case "${{ inputs.action || 'audit' }}" in
            audit)
              audit_rbac_permissions "${{ matrix.environment }}"
              ;;
            cleanup)
              cleanup_unused_rbac "${{ matrix.environment }}"
              ;;
            validate)
              validate_rbac_least_privilege "${{ matrix.environment }}"
              ;;
          esac

      - name: Generate RBAC Report
        if: always()
        run: |
          mkdir -p rbac-reports
          cat > rbac-reports/rbac-${{ matrix.environment }}.md << EOF
          # RBAC Audit Report

          **Environment**: ${{ matrix.environment }}
          **Date**: $(date)
          **Action**: ${{ inputs.action || 'audit' }}

          ## Service Accounts Audited

          $(kubectl get serviceaccounts --all-namespaces | grep -v default | head -20)

          ## Cluster Roles

          $(kubectl get clusterroles | grep coder)

          ## Recommendations

          - Review service account permissions quarterly
          - Remove unused RBAC resources
          - Ensure least privilege principle compliance
          EOF

      - name: Upload RBAC Report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: rbac-report-${{ matrix.environment }}
          path: rbac-reports/

Implementation Steps:

1. Deploy RBAC module in infrastructure phase with environment-specific policies
2. Update Coder deployment to use least privilege service accounts
3. Create RBAC audit scripts with remote state integration
4. Set up automated RBAC validation workflows
5. Configure regular cleanup of unused RBAC resources

Validation Procedures:

• Test service account permissions across environments
• Verify least privilege compliance in production
• Confirm unused RBAC resources are properly identified and cleaned up

---

UC-P0-005: Network Security Hardening

User Story: As a network security engineer, I want to implement comprehensive network policies and traffic isolation to prevent lateral movement and unauthorized access.

Current State: Basic networking exists but lacks network policies, traffic isolation, and comprehensive security monitoring.

Remote State Integration:

• Infrastructure phase deploys network policies, stores network configuration in remote state
• Coder phase accesses network settings from infrastructure remote state
• Scripts validate network security using remote state network metadata

Acceptance Criteria:

• Network policies enforce zero-trust micro-segmentation
• All inter-pod communication is explicitly allowed or denied
• Network traffic monitoring and anomaly detection implemented
• Environment-specific network security (stricter in production)

Cross-Technology Implementation:

Terraform Infrastructure Phase (modules/networking/security.tf):

# Environment-specific network security configuration
locals {
  network_security_config = {
    dev = {
      default_deny_all = false
      allow_cross_namespace = true
      enable_ingress_from_internet = true
      enable_egress_to_internet = true
      monitoring_level = "basic"
    }
    staging = {
      default_deny_all = true
      allow_cross_namespace = false
      enable_ingress_from_internet = true
      enable_egress_to_internet = true
      monitoring_level = "enhanced"
    }
    prod = {
      default_deny_all = true
      allow_cross_namespace = false
      enable_ingress_from_internet = false  # Only through load balancer
      enable_egress_to_internet = false     # Explicit allow lists only
      monitoring_level = "comprehensive"
    }
  }

  security_config = local.network_security_config[var.environment]
}

# Default deny-all network policy
resource "kubernetes_network_policy" "default_deny_all" {
  count = local.security_config.default_deny_all ? 1 : 0

  metadata {
    name      = "default-deny-all"
    namespace = "coder"
  }

  spec {
    pod_selector {}
    policy_types = ["Ingress", "Egress"]
  }
}

# Coder application network policy
resource "kubernetes_network_policy" "coder_app" {
  metadata {
    name      = "coder-app"
    namespace = "coder"
  }

  spec {
    pod_selector {
      match_labels = {
        "app.kubernetes.io/name" = "coder"
      }
    }

    policy_types = ["Ingress", "Egress"]

    # Allow ingress from load balancer
    ingress {
      from {
        namespace_selector {
          match_labels = {
            name = "ingress-nginx"
          }
        }
      }
      ports {
        protocol = "TCP"
        port     = "7080"
      }
    }

    # Allow ingress from workspace namespaces
    ingress {
      from {
        namespace_selector {
          match_labels = {
            "coder.com/workspace" = "true"
          }
        }
      }
    }

    # Allow egress to database
    egress {
      to {
        namespace_selector {
          match_labels = {
            name = "database"
          }
        }
      }
      ports {
        protocol = "TCP"
        port     = "5432"
      }
    }

    # Allow DNS resolution
    egress {
      to {}
      ports {
        protocol = "UDP"
        port     = "53"
      }
    }

    # Environment-specific egress rules
    dynamic "egress" {
      for_each = local.security_config.enable_egress_to_internet ? [1] : []
      content {
        to {}  # Allow all outbound traffic
      }
    }
  }
}

# Workspace network policies
resource "kubernetes_network_policy" "workspace_isolation" {
  metadata {
    name      = "workspace-isolation"
    namespace = "coder-workspaces"
  }

  spec {
    pod_selector {}
    policy_types = ["Ingress", "Egress"]

    # Allow ingress from Coder application
    ingress {
      from {
        namespace_selector {
          match_labels = {
            name = "coder"
          }
        }
      }
    }

    # Allow egress to Coder application
    egress {
      to {
        namespace_selector {
          match_labels = {
            name = "coder"
          }
        }
      }
    }

    # DNS resolution
    egress {
      to {}
      ports {
        protocol = "UDP"
        port     = "53"
      }
    }

    # Environment-specific cross-workspace communication
    dynamic "ingress" {
      for_each = local.security_config.allow_cross_namespace ? [1] : []
      content {
        from {
          namespace_selector {
            match_labels = {
              "coder.com/workspace" = "true"
            }
          }
        }
      }
    }
  }
}

# Cilium network policies for advanced features (if using Cilium CNI)
resource "kubectl_manifest" "cilium_network_policy" {
  count = var.enable_cilium_policies ? 1 : 0

  yaml_body = yamlencode({
    apiVersion = "cilium.io/v2"
    kind       = "CiliumNetworkPolicy"
    metadata = {
      name      = "coder-l7-policy"
      namespace = "coder"
    }
    spec = {
      podSelector = {
        matchLabels = {
          "app.kubernetes.io/name" = "coder"
        }
      }
      ingress = [
        {
          fromEndpoints = [
            {
              matchLabels = {
                "app" = "nginx-ingress"
              }
            }
          ]
          toPorts = [
            {
              ports = [
                {
                  port     = "7080"
                  protocol = "TCP"
                }
              ]
              rules = {
                http = [
                  {
                    method = "GET"
                    path   = "/api/v2/.*"
                  },
                  {
                    method = "POST"
                    path   = "/api/v2/login"
                  }
                ]
              }
            }
          ]
        }
      ]
    }
  })
}

# Output network security configuration for remote state
output "network_security_config" {
  description = "Network security configuration"
  value = {
    default_deny_all = local.security_config.default_deny_all
    cross_namespace_allowed = local.security_config.allow_cross_namespace
    monitoring_level = local.security_config.monitoring_level
    policies_created = [
      "default-deny-all",
      "coder-app",
      "workspace-isolation"
    ]
  }
}

output "network_topology" {
  description = "Network topology information"
  value = {
    coder_namespace = "coder"
    workspace_namespace = "coder-workspaces"
    ingress_namespace = "ingress-nginx"
    allowed_ingress_ports = ["7080"]
    allowed_egress_ports = ["5432", "53"]
  }
}

Terraform Coder Phase (modules/coder-deployment/networking.tf):

# Access network configuration from infrastructure remote state
data "terraform_remote_state" "infra" {
  # ... backend configuration
}

locals {
  network_config = data.terraform_remote_state.infra.outputs.network_security_config
  network_topology = data.terraform_remote_state.infra.outputs.network_topology
}

# Coder service with network policy labels
resource "kubernetes_service" "coder" {
  metadata {
    name      = "coder"
    namespace = local.network_topology.coder_namespace
    labels = merge(var.labels, {
      "app.kubernetes.io/name" = "coder"
      "network-policy"         = "coder-app"
    })
  }

  spec {
    selector = {
      "app.kubernetes.io/name" = "coder"
    }

    port {
      name        = "http"
      port        = 80
      target_port = 7080
      protocol    = "TCP"
    }

    type = "ClusterIP"
  }
}

# Workspace template with network security constraints
resource "coder_template" "secure_workspace" {
  name         = "secure-workspace"
  description  = "Security-hardened workspace with network policies"

  template = templatefile("${path.module}/templates/secure-workspace.tf.tpl", {
    network_config = local.network_config
    network_topology = local.network_topology
    cross_namespace_allowed = local.network_config.cross_namespace_allowed
  })
}

# Network monitoring configuration
resource "kubernetes_config_map" "network_monitoring" {
  count = local.network_config.monitoring_level == "comprehensive" ? 1 : 0

  metadata {
    name      = "network-monitoring-config"
    namespace = "monitoring"
  }

  data = {
    "cilium-config.yaml" = yamlencode({
      enableNetworkPolicy = true
      enableTracing = true
      monitor = {
        aggregation = "maximum"
        interval = "5s"
      }
      flowlog = {
        enabled = true
        level   = "info"
      }
    })
  }
}

Bash Script Integration (scripts/security/network-security.sh):

#!/bin/bash
# Network security validation with remote state integration

validate_network_policies() {
    local env="$1"

    log "Validating network policies for environment: $env"

    # Get network configuration from infrastructure remote state
    cd "environments/${env}/infra"
    terraform init -input=false >/dev/null 2>&1

    local network_config
    network_config=$(terraform output -json network_security_config 2>/dev/null)

    if [[ -z "$network_config" ]]; then
        log_error "Cannot retrieve network configuration from remote state"
        return 1
    fi

    local policies_created
    policies_created=$(echo "$network_config" | jq -r '.policies_created[]')

    # Validate each network policy exists
    while IFS= read -r policy; do
        if kubectl get networkpolicy "$policy" -n coder >/dev/null 2>&1; then
            log_success "Network policy exists: $policy"

            # Validate policy configuration
            validate_network_policy_config "$policy" "$network_config"
        else
            log_error "Network policy missing: $policy"
        fi
    done <<< "$policies_created"

    # Test network connectivity
    test_network_connectivity "$env" "$network_config"
}

validate_network_policy_config() {
    local policy="$1"
    local config="$2"

    local policy_spec
    policy_spec=$(kubectl get networkpolicy "$policy" -n coder -o json)

    # Check if default deny policy exists when required
    local default_deny_all
    default_deny_all=$(echo "$config" | jq -r '.default_deny_all')

    if [[ "$default_deny_all" == "true" && "$policy" == "default-deny-all" ]]; then
        # Validate deny-all policy configuration
        local policy_types
        policy_types=$(echo "$policy_spec" | jq -r '.spec.policyTypes[]')

        if [[ "$policy_types" != *"Ingress"* ]] || [[ "$policy_types" != *"Egress"* ]]; then
            log_error "Default deny-all policy does not cover both Ingress and Egress"
            return 1
        fi

        log_success "Default deny-all policy correctly configured"
    fi
}

test_network_connectivity() {
    local env="$1"
    local config="$2"

    log "Testing network connectivity constraints..."

    # Deploy test pods for connectivity testing
    kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: network-test-coder
  namespace: coder
  labels:
    app.kubernetes.io/name: coder
spec:
  containers:
  - name: test
    image: nicolaka/netshoot:latest
    command: ["sleep", "300"]
  restartPolicy: Never
---
apiVersion: v1
kind: Pod
metadata:
  name: network-test-workspace
  namespace: coder-workspaces
  labels:
    coder.com/workspace: "true"
spec:
  containers:
  - name: test
    image: nicolaka/netshoot:latest
    command: ["sleep", "300"]
  restartPolicy: Never
EOF

    # Wait for pods to be ready
    kubectl wait --for=condition=Ready pod/network-test-coder -n coder --timeout=60s
    kubectl wait --for=condition=Ready pod/network-test-workspace -n coder-workspaces --timeout=60s

    # Test allowed connections
    log "Testing allowed Coder to workspace communication..."
    if kubectl exec -n coder network-test-coder -- nc -zv network-test-workspace.coder-workspaces.svc.cluster.local 22 2>/dev/null; then
        log_success "Coder to workspace communication allowed (expected)"
    else
        log_error "Coder to workspace communication blocked (unexpected)"
    fi

    # Test denied connections (if cross-namespace is disabled)
    local cross_namespace_allowed
    cross_namespace_allowed=$(echo "$config" | jq -r '.cross_namespace_allowed')

    if [[ "$cross_namespace_allowed" == "false" ]]; then
        log "Testing denied cross-namespace communication..."
        if ! kubectl exec -n coder-workspaces network-test-workspace -- nc -zv kubernetes.default.svc.cluster.local 443 2>/dev/null; then
            log_success "Cross-namespace communication properly denied"
        else
            log_warning "Cross-namespace communication unexpectedly allowed"
        fi
    fi

    # Cleanup test pods
    kubectl delete pod network-test-coder -n coder --ignore-not-found
    kubectl delete pod network-test-workspace -n coder-workspaces --ignore-not-found
}

monitor_network_traffic() {
    local env="$1"
    local duration="${2:-60}"  # Default 60 seconds

    log "Monitoring network traffic for $duration seconds..."

    # Get monitoring level from remote state
    cd "environments/${env}/infra"
    local monitoring_level
    monitoring_level=$(terraform output -json network_security_config 2>/dev/null | jq -r '.monitoring_level')

    case "$monitoring_level" in
        "comprehensive")
            # Use Cilium Hubble for comprehensive monitoring
            if command -v hubble >/dev/null 2>&1; then
                log "Starting Cilium Hubble monitoring..."
                hubble observe --namespace coder --follow --timeout "${duration}s" > network-traffic-${env}.log &
                local hubble_pid=$!

                sleep "$duration"
                kill $hubble_pid 2>/dev/null

                # Analyze traffic patterns
                if [[ -f "network-traffic-${env}.log" ]]; then
                    local denied_flows
                    denied_flows=$(grep "DENIED" "network-traffic-${env}.log" | wc -l)

                    if [[ "$denied_flows" -gt 0 ]]; then
                        log_warning "Found $denied_flows denied network flows"
                        log "Recent denied flows:"
                        grep "DENIED" "network-traffic-${env}.log" | tail -5
                    else
                        log_success "No denied network flows detected"
                    fi
                fi
            else
                log_warning "Hubble not available, falling back to basic monitoring"
                monitor_basic_network_traffic "$env" "$duration"
            fi
            ;;
        "enhanced"|"basic")
            monitor_basic_network_traffic "$env" "$duration"
            ;;
    esac
}

monitor_basic_network_traffic() {
    local env="$1"
    local duration="$2"

    log "Starting basic network monitoring using netstat..."

    # Monitor network connections from Coder pods
    local coder_pods
    coder_pods=$(kubectl get pods -n coder -l app.kubernetes.io/name=coder -o jsonpath='{.items[*].metadata.name}')

    for pod in $coder_pods; do
        log "Network connections for pod: $pod"
        kubectl exec -n coder "$pod" -- netstat -tuln 2>/dev/null | head -10
    done
}

audit_network_security() {
    local env="$1"

    log "Auditing network security configuration for environment: $env"

    # Check for pods without network policies
    local unprotected_namespaces
    unprotected_namespaces=$(kubectl get namespaces -o json | \
        jq -r '.items[] | select(.metadata.name | test("kube-|default") | not) | .metadata.name' | \
        while read -r ns; do
            local policy_count
            policy_count=$(kubectl get networkpolicy -n "$ns" --no-headers 2>/dev/null | wc -l)
            if [[ "$policy_count" -eq 0 ]]; then
                echo "$ns"
            fi
        done)

    if [[ -n "$unprotected_namespaces" ]]; then
        log_warning "Namespaces without network policies:"
        echo "$unprotected_namespaces" | while read -r ns; do
            log_warning "  - $ns"
        done
    else
        log_success "All namespaces have network policies"
    fi

    # Check for overly permissive policies
    log "Checking for overly permissive network policies..."
    kubectl get networkpolicy --all-namespaces -o json | \
        jq -r '.items[] | select(.spec.ingress[]?.from == [] or .spec.egress[]?.to == []) | "\(.metadata.namespace)/\(.metadata.name)"' | \
        while read -r policy; do
            log_warning "Overly permissive network policy: $policy"
        done
}

GitHub Workflow Integration (.github/workflows/network-security.yml):

name: Network Security Validation

on:
  schedule:
    - cron: '0 4 * * *'  # Daily at 4 AM
  workflow_dispatch:
    inputs:
      environment:
        required: true
        type: choice
        options: [dev, staging, prod]
      action:
        required: true
        type: choice
        options: [validate, monitor, audit]

jobs:
  network-security:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        environment: ${{ github.event_name == 'workflow_dispatch' && fromJSON(format('["{0}"]', inputs.environment)) || fromJSON('["dev", "staging", "prod"]') }}

    steps:
      - name: Checkout
        uses: actions/checkout@v5

      - name: Setup Tools
        run: |
          # Install network security tools
          curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
          chmod +x kubectl
          sudo mv kubectl /usr/local/bin/

          # Install Cilium CLI for advanced monitoring
          curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz
          tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
          rm cilium-linux-amd64.tar.gz

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: "~1.12.0"
          terraform_wrapper: false

      - name: Download Backend Configuration
        uses: actions/download-artifact@v5
        with:
          name: backend-config-${{ matrix.environment }}-infra-${{ github.run_id }}
          path: environments/${{ matrix.environment }}/infra

      - name: Download Kubeconfig
        uses: actions/download-artifact@v5
        with:
          name: kubeconfig-${{ matrix.environment }}-infra
          path: .

      - name: Setup Kubernetes Access
        run: |
          mkdir -p ~/.kube
          cp kubeconfig-${{ matrix.environment }} ~/.kube/config
          chmod 600 ~/.kube/config

          # Verify connectivity
          kubectl cluster-info

      - name: Execute Network Security Action
        env:
          SCW_ACCESS_KEY: ${{ secrets.SCW_ACCESS_KEY }}
          SCW_SECRET_KEY: ${{ secrets.SCW_SECRET_KEY }}
        run: |
          source scripts/security/network-security.sh

          case "${{ inputs.action || 'validate' }}" in
            validate)
              validate_network_policies "${{ matrix.environment }}"
              ;;
            monitor)
              monitor_network_traffic "${{ matrix.environment }}" 120
              ;;
            audit)
              audit_network_security "${{ matrix.environment }}"
              ;;
          esac

      - name: Generate Network Security Report
        if: always()
        run: |
          mkdir -p network-reports
          cat > network-reports/network-security-${{ matrix.environment }}.md << EOF
          # Network Security Report

          **Environment**: ${{ matrix.environment }}
          **Date**: $(date)
          **Action**: ${{ inputs.action || 'validate' }}

          ## Network Policies

          $(kubectl get networkpolicy --all-namespaces)

          ## Security Configuration

          $(cd environments/${{ matrix.environment }}/infra && terraform output network_security_config 2>/dev/null || echo "Configuration not available")

          ## Recommendations

          - Review network policies quarterly
          - Monitor denied network flows
          - Ensure least privilege network access
          - Test connectivity constraints regularly
          EOF

      - name: Upload Network Reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: network-report-${{ matrix.environment }}
          path: network-reports/

      - name: Upload Traffic Logs
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: network-traffic-${{ matrix.environment }}
          path: "*.log"

Implementation Steps:

1. Deploy network security module in infrastructure phase with environment-specific policies
2. Update Coder deployment with network policy labels and constraints
3. Create network validation scripts with remote state integration
4. Set up automated network security monitoring workflows
5. Configure traffic analysis and anomaly detection

Validation Procedures:

• Test network policies block unauthorized connections
• Verify allowed traffic flows work correctly
• Confirm network monitoring captures security events
• Validate environment-specific network restrictions

---

Summary

This V3 implementation plan addresses the critical security issues identified in the Terraform Security Review while properly accounting for the remote state architecture using Scaleway Object Storage. Each use case demonstrates proper cross-technology integration patterns:

1. Terraform creates secure infrastructure and stores configuration in remote state
2. Bash Scripts access configuration via terraform output commands with proper backend initialization
3. GitHub Workflows coordinate deployment phases and manage backend configurations

The remote state integration ensures that security configurations are centrally managed, consistently applied, and properly accessible across all tools and environments while maintaining the two-phase deployment architecture's benefits.

---

Priority 1 (P1) - High Priority Security Improvements

UC-P1-001: State Encryption and Access Control

User Story: As a security administrator, I want Terraform state encrypted at rest and in transit with proper access controls.

Remote State Implementation: Enhanced Scaleway Object Storage bucket encryption and IAM policies for state access.

UC-P1-002: Audit Logging and Monitoring

User Story: As a compliance officer, I want comprehensive audit logs of all infrastructure changes and security events.

Remote State Integration: Audit logs stored in remote state, accessible via Terraform outputs for compliance reporting.

UC-P1-003: Automated Security Scanning

User Story: As a DevSecOps engineer, I want automated security scanning of infrastructure and container images.

Cross-Technology Pattern: Terraform deploys scanning tools, bash scripts execute scans, GitHub workflows aggregate results.

---

Priority 2 (P2) - Medium Priority Enhancements

UC-P2-001: Resource Tagging and Cost Management

User Story: As a financial controller, I want comprehensive resource tagging for cost allocation and management.

Remote State Usage: Cost allocation data stored in remote state for cross-environment reporting.

UC-P2-002: Backup and Disaster Recovery

User Story: As a platform administrator, I want automated backup and disaster recovery procedures.

Remote State Coordination: Backup metadata and recovery procedures coordinated through remote state.

UC-P2-003: Performance Monitoring and Optimization

User Story: As a site reliability engineer, I want performance monitoring and automated optimization.

Implementation Pattern: Monitoring deployed via Terraform, metrics accessed via bash scripts, optimization triggered by GitHub workflows.

---

Implementation Roadmap

Phase 1: P0 Critical Security (Weeks 1-4)

1. Input validation and command injection prevention
2. Scaleway Secret Manager integration
3. Pod Security Standards implementation
4. RBAC least privilege enforcement
5. Network security hardening

Phase 2: P1 High Priority (Weeks 5-8)

1. State encryption and access control
2. Audit logging and monitoring
3. Automated security scanning

Phase 3: P2 Medium Priority (Weeks 9-12)

1. Resource tagging and cost management
2. Backup and disaster recovery
3. Performance monitoring and optimization

Remote State Architecture Benefits

Centralized Configuration Management:

• All security settings stored in Scaleway Object Storage
• Cross-phase data sharing via terraform_remote_state data sources
• Consistent configuration access across all tools

Enhanced Security:

• State files encrypted at rest in Scaleway Object Storage
• Access controlled via Scaleway IAM policies
• No sensitive data in local state files

Improved Collaboration:

• Teams can access shared state remotely
• GitHub Actions coordinate deployments centrally
• Scripts operate on consistent remote state data

Operational Excellence:

• State versioning and backup built into Scaleway Object Storage
• Cross-environment state isolation
• Audit trail of all state modifications

This V3 implementation plan provides a comprehensive security enhancement roadmap that leverages the remote state architecture for consistent, secure, and collaborative infrastructure management.

[pacphi]

Infrastructure Security and Optimization Implementation Proposal

Executive Summary

Following a comprehensive review of the Terraform infrastructure code, GitHub Actions workflows, and bash scripts in the bootstrap-coder-on-scaleway repository, this proposal outlines critical security vulnerabilities and optimization opportunities that require immediate attention.

The review identified 17 critical security issues, 23 high-priority improvements, and 15 optimization opportunities across the codebase. The most pressing concerns involve network security configurations that expose critical infrastructure components to the internet, credential management vulnerabilities, and CI/CD supply chain risks.

Critical Security Vulnerabilities Summary

Infrastructure Layer (Terraform)

• 🚨 CRITICAL: Unrestricted network access (0.0.0.0/0) for SSH, Kubernetes API, and database
• 🚨 CRITICAL: Overly permissive RBAC configurations allowing cluster-wide deletions
• 🔴 HIGH: Sensitive credentials stored in Terraform state files
• 🔴 HIGH: Missing pod security contexts enabling privilege escalation

CI/CD Pipeline (GitHub Actions)

• 🚨 CRITICAL: Unpinned third-party actions vulnerable to supply chain attacks
• 🔴 HIGH: Direct secret exposure without OIDC authentication
• 🔴 HIGH: Overly broad workflow permissions

Automation Scripts (Bash)

• 🚨 CRITICAL: Command injection vulnerabilities in database backup operations
• 🔴 HIGH: Credential exposure in process lists and logs
• 🔴 HIGH: Path traversal vulnerabilities in file operations

Implementation Roadmap

Phase 1: Critical Security Remediation (Week 1)

Day 1-2: Network Security Hardening

Owner: Infrastructure Team
Priority: P0

1. Restrict Network Access

   # File: modules/networking/main.tf
   # Replace all instances of 0.0.0.0/0 with specific CIDR ranges
   
   resource "scaleway_instance_security_group_rules" "cluster" {
     security_group_id = scaleway_instance_security_group.cluster.id
   
     # SSH access - restrict to management network
     inbound_rule {
       action   = "accept"
       protocol = "TCP"
       port     = 22
       ip_range = var.management_cidr  # e.g., "203.0.113.0/24"
     }
   
     # Kubernetes API - restrict to VPN/bastion
     inbound_rule {
       action   = "accept"
       protocol = "TCP"
       port     = 6443
       ip_range = var.trusted_cidr  # e.g., "10.0.0.0/8"
     }
   }

2. Database ACL Restrictions

   # File: modules/postgresql/main.tf
   resource "scaleway_rdb_acl" "postgresql_acl" {
     instance_id = scaleway_rdb_instance.postgresql.id
   
     acl_rules {
       ip          = var.vpc_cidr  # Only VPC access
       description = "VPC network access only"
     }
   }

Day 3-4: RBAC and Pod Security

Owner: Security Team
Priority: P0

1. Implement Least Privilege RBAC

   # File: modules/security/rbac-config.yaml
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: coder-restricted
     namespace: coder
   rules:
   - apiGroups: [""]
     resources: ["pods", "pods/log"]
     verbs: ["get", "list", "create", "update", "patch", "watch"]
   - apiGroups: [""]
     resources: ["secrets"]
     verbs: ["get", "list"]  # Read-only for secrets

2. Enforce Pod Security Standards

   # File: modules/coder-deployment/security-context.yaml
   securityContext:
     runAsNonRoot: true
     runAsUser: 1000
     runAsGroup: 1000
     fsGroup: 1000
     allowPrivilegeEscalation: false
     readOnlyRootFilesystem: true
     seccompProfile:
       type: RuntimeDefault
     capabilities:
       drop: ["ALL"]

Day 5: CI/CD Security Hardening

Owner: DevOps Team
Priority: P0

1. Pin GitHub Actions to SHA

   # File: .github/workflows/*.yml
   # Update all action references
   - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
   - uses: hashicorp/setup-terraform@633666f66e0061ca3b725c73b2ec20cd13a8fdd1 # v2.0.3

2. Implement Dependabot Configuration

   # File: .github/dependabot.yml
   version: 2
   updates:
     - package-ecosystem: "github-actions"
       directory: "/"
       schedule:
         interval: "weekly"
       reviewers:
         - "security-team"
   
     - package-ecosystem: "terraform"
       directory: "/"
       schedule:
         interval: "weekly"

Phase 2: High-Priority Security (Week 2)

Day 6-7: Scaleway Secret Manager Integration

Owner: Security Team
Priority: P1

📋 DETAILED IMPLEMENTATION: See #33 (comment) for comprehensive Scaleway Secret Manager integration with External Secrets Operator.

Key Implementation Points:

1. Deploy Scaleway Secret Manager with External Secrets Operator

   • Enterprise-grade AES-256 KMS encryption
   • Kubernetes-native secret synchronization
   • Regional deployment with high availability
   • Automated secret rotation capabilities

2. Database Credential Migration

   # Replace direct credential outputs with secret references
   output "database_secret_name" {
     description = "Kubernetes secret containing database credentials"
     value       = "database-credentials"
   }
   
   output "database_secret_namespace" {
     description = "Namespace containing database secret"
     value       = "coder"
   }

3. Template Secret Integration

   • Template-specific secret scoping (AI API keys, GitHub tokens)
   • Namespace-scoped RBAC for secret access
   • Automatic injection into workspace pods
   • Centralized audit logging and compliance

Security Benefits:

• ✅ Zero credentials in Terraform state files
• ✅ SOC2 Type 2 compliant secret storage
• ✅ Automated secret rotation and lifecycle management
• ✅ Regional data residency compliance
• ✅ Centralized audit trail for all secret operations

Cost Impact: €15-25/month per environment (ROI: 300-500% within 6 months)

Day 8-9: Script Security Fixes

Owner: Platform Team
Priority: P1

1. Fix Command Injection Vulnerabilities

   # File: scripts/lifecycle/backup.sh
   # Secure database backup implementation
   backup_database() {
       local db_host="$1"
       local db_user="$2"
       local db_name="$3"
   
       # Use temporary file for credentials
       local temp_pgpass=$(mktemp)
       trap "rm -f '$temp_pgpass'" EXIT
   
       # Write credentials securely
       {
           echo "$db_host:5432:$db_name:$db_user:$db_pass"
       } > "$temp_pgpass"
       chmod 600 "$temp_pgpass"
   
       # Execute backup with secure credential handling
       PGPASSFILE="$temp_pgpass" pg_dump \
           -h "$db_host" \
           -U "$db_user" \
           -d "$db_name" \
           --clean \
           --create > "${backup_dir}/database.sql"
   }

2. Implement Input Validation Library

   # File: scripts/lib/validation.sh
   validate_environment() {
       local env="$1"
       if [[ ! "$env" =~ ^(dev|staging|prod)$ ]]; then
           log ERROR "Invalid environment: $env"
           return 1
       fi
   }
   
   validate_domain() {
       local domain="$1"
       local regex='^[a-zA-Z0-9]([a-zA-Z0-9\-]*[a-zA-Z0-9])?(\.[a-zA-Z0-9]([a-zA-Z0-9\-]*[a-zA-Z0-9])?)*$'
       if [[ ! "$domain" =~ $regex ]]; then
           log ERROR "Invalid domain: $domain"
           return 1
       fi
   }

Day 10: Workflow Permissions

Owner: DevOps Team
Priority: P1

1. Apply Least Privilege to Workflows

   # File: .github/workflows/deploy-environment.yml
   name: Deploy Environment
   
   permissions: {}  # Default to no permissions
   
   jobs:
     deploy:
       permissions:
         contents: read        # Only read repository
         id-token: write      # For OIDC authentication
       steps:
         - uses: actions/checkout@v4

Phase 3: Optimization Implementation (Week 3-4)

Cost Optimization

Owner: FinOps Team
Priority: P2

1. Storage Optimization

   # File: environments/dev/infra/main.tf
   module "cluster" {
     source = "../../../modules/scaleway-cluster"
   
     # Optimize storage types
     root_volume_type = "b_ssd"  # Standard SSD for dev
     root_volume_size = 50        # Reduced from 100GB
   }

2. Auto-scaling Refinement

   # File: environments/prod/infra/main.tf
   auto_scale = {
     min_size                  = 3   # Reduced from 5
     max_size                  = 10  # Reduced from 15
     scale_down_delay_after_add = "5m"
     scale_down_unneeded_time  = "10m"
   }

CI/CD Performance

Owner: DevOps Team
Priority: P2

1. Implement Caching Strategy

   # File: .github/workflows/_caching.yml
   - name: Cache Terraform Providers
     uses: actions/cache@v4
     with:
       path: |
         ~/.terraform.d/plugins
         .terraform/providers
       key: terraform-${{ runner.os }}-${{ hashFiles('**/.terraform.lock.hcl') }}
       restore-keys: |
         terraform-${{ runner.os }}-
   
   - name: Cache Tools
     uses: actions/cache@v4
     with:
       path: |
         /usr/local/bin/kubectl
         /usr/local/bin/helm
       key: tools-${{ runner.os }}-${{ hashFiles('.tool-versions') }}

2. Parallel Job Execution

   # File: .github/workflows/validate-templates.yml
   jobs:
     discover:
       outputs:
         template-matrix: ${{ steps.discover.outputs.matrix }}
   
     validate:
       needs: discover
       strategy:
         matrix: ${{ fromJson(needs.discover.outputs.template-matrix) }}
         max-parallel: 5
       steps:
         - name: Validate Template
           run: ./scripts/validate-template.sh ${{ matrix.template }}

Phase 4: Advanced Security and Monitoring (Week 5-6)

OIDC Authentication

Owner: Security Team
Priority: P2

1. Implement OIDC for Scaleway

   # File: .github/workflows/deploy-environment.yml
   - name: Configure Scaleway OIDC
     uses: scaleway/setup-scaleway@v2
     with:
       oidc-audience: ${{ vars.SCALEWAY_OIDC_AUDIENCE }}
       oidc-provider: ${{ vars.SCALEWAY_OIDC_PROVIDER }}
       region: fr-par

Security Scanning Integration

Owner: Security Team
Priority: P2

1. Add Security Scanning

   # File: .github/workflows/security-scan.yml
   name: Security Scanning
   
   on:
     pull_request:
     push:
       branches: [main]
   
   jobs:
     terraform-security:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v4
   
         - name: Run Checkov
           uses: bridgecrewio/checkov-action@master
           with:
             directory: .
             framework: terraform
             output_format: sarif
   
         - name: Upload SARIF
           uses: github/codeql-action/upload-sarif@v2
           with:
             sarif_file: results.sarif

Success Metrics

Security Metrics

• Zero publicly exposed critical services (SSH, K8s API, Database)
• 100% of GitHub Actions pinned to SHA hashes
• Zero credentials in Terraform state files
• 100% of pods running with security contexts
• Zero high/critical vulnerabilities in security scans

Performance Metrics

• 40-50% reduction in CI/CD pipeline runtime
• 30% reduction in infrastructure costs
• 80%+ cache hit rate in workflows
• 2-3 minute automated recovery from failed deployments

Operational Metrics

• 100% input validation coverage in scripts
• Zero command injection vulnerabilities
• 100% of secrets managed externally
• Automated security scanning on all PRs

Risk Mitigation

Implementation Risks

1. Service Disruption: Implement changes in dev → staging → prod progression
2. Breaking Changes: Comprehensive testing before each phase
3. Team Capacity: Phased approach allows gradual implementation

Rollback Strategy

1. All changes tracked in version control
2. Terraform state backups before each change
3. Blue-green deployment patterns for critical changes
4. Automated rollback triggers on failure metrics

Resource Requirements

Team Allocation

• Security Team: 2 engineers for 2 weeks (Phase 1-2)
• Infrastructure Team: 1 engineer for 1 week (Phase 1)
• DevOps Team: 2 engineers for 3 weeks (Phase 1-3)
• Platform Team: 1 engineer for 1 week (Phase 2)

Tools and Services

• HashiCorp Vault or AWS Secrets Manager for credential management
• Dependabot for dependency management (free with GitHub)
• Checkov/Trivy for security scanning (open source)
• GitHub Advanced Security (optional, for enhanced scanning)

Timeline Summary

Phase	Duration	Focus Area	Priority
Phase 1	Week 1	Critical Security	P0
Phase 2	Week 2	High-Priority Security	P1
Phase 3	Week 3-4	Optimization	P2
Phase 4	Week 5-6	Advanced Security	P2

Conclusion

This proposal addresses critical security vulnerabilities that pose immediate risks to the infrastructure. The phased implementation approach ensures systematic remediation while maintaining service availability. Priority should be given to Phase 1 activities, particularly network security hardening, which addresses the most severe vulnerabilities.

The optimization opportunities identified will result in significant cost savings and performance improvements, making the infrastructure more efficient and maintainable. Regular security audits and automated scanning will ensure ongoing compliance and security posture improvement.

Appendices

A. Detailed Finding Reports

• Terraform Security Review
• GitHub Actions Review
• Bash Scripts Review
• Scaleway Secrets Management Integration ⭐ NEW

B. Security Checklist

• Network access restricted to specific CIDRs
• Database ACLs configured for VPC-only access
• RBAC policies follow least privilege
• Pod security contexts enforced
• GitHub Actions pinned to SHA hashes
• Scaleway Secret Manager deployed and configured ⭐
• External Secrets Operator integrated with Kubernetes ⭐
• Database credentials migrated from Terraform state ⭐
• Template secrets configured with namespace scoping ⭐
• Credentials removed from state files
• Input validation implemented in all scripts
• Command injection vulnerabilities fixed
• Security scanning integrated in CI/CD

C. Cost Optimization Checklist

• Storage types optimized per environment
• Auto-scaling ranges adjusted
• Caching implemented in workflows
• Parallel job execution enabled
• Resource quotas configured
• Cost alerts configured

---

Document Version: 1.0
Date: 2025-08-19
Authors: Security Review Team
Status: DRAFT - Pending Review