Skip to content

security.md

Latest commit

 

History

History
757 lines (535 loc) · 20.5 KB

security.md

File metadata and controls

757 lines (535 loc) · 20.5 KB

Security Baseline Guide — Accellens

Version: 1.0 Date: November 10, 2025

1. Overview

This guide describes the security baseline configuration for Accellens in all environments: AI safety filters, PII redaction, audit logging, security scanning, and Swagger UI security.

2. Safety Filters for AI Prompts

2.1 Components

PII Detection and Redaction:

  • Uses Presidio Analyzer for PII detection
  • Automatic redaction before sending to LLM
  • Supported types: email, phone, SSN, credit card, IP addresses

Toxicity Detection:

  • OpenAI Moderation API (primary, if available)
  • Detoxify (fallback)
  • Keyword-based check (fallback)

2.2 Configuration

# In config.py
ai_enable_safety_filters: bool = True
openai_api_key: str | None = None  # For moderation API

2.3 Usage

Safety filters are automatically applied in AIProviderRouter:

from ai.src.router import AIProviderRouter

router = AIProviderRouter(enable_safety=True)
# PII and toxicity checks are performed automatically

2.4 Violation Logging

All safety violations are logged with full details:

logger.warning("PII detected and redacted in prompt", extra={"entities": len(pii_entities)})
logger.warning("Toxicity detected in prompt", extra={"method": "openai_moderation", "categories": {...}})

3. PII Redaction Middleware

3.1 Gateway Middleware

Middleware for automatic PII redaction from request/response:

import { piiRedactionMiddleware } from '@gateway/middleware/pii-redaction.middleware';

// Registration in Fastify
fastify.addHook('onRequest', piiRedactionMiddleware);

3.2 Python Services Middleware

A similar pipeline is implemented for all FastAPI microservices:

  • apps/services/middleware/pii_redaction.py intercepts POST/PUT/PATCH/DELETE, redacts body and query, stores result in request.state.
  • apps/services/middleware/audit_logging.py creates entries in audit_logs after successful write operations, using redacted payload + list of detected PII types, request id, user/org from headers (X-Actor-Id, X-Organization-Id) and IP address.

Thus, services meet requirements 1.3.12/1.3.13 for multi-tenant isolation and security baseline.

3.3 Detected PII Types

  • Email addresses
  • Phone numbers
  • Credit card numbers
  • SSN
  • IP addresses (optional)

3.4 Usage

Middleware automatically:

  • Detects PII in request body
  • Detects PII in query parameters
  • Logs detected PII types
  • Stores information for audit logging

4. Secrets Management

4.1 Dev Environment

Local secrets:

  • Stored in .env files
  • Gitignored (added to .gitignore)
  • Minimal security requirements

Recommendations:

  • Do not commit .env files
  • Use .env.example as a template
  • Rotate secrets when necessary
  • Use secrets_provider=env (default in apps/services/config.py)

4.2 Test Environment

SOPS + Sealed Secrets:

# Install SOPS
brew install sops  # macOS
apt-get install sops  # Linux

# Install Sealed Secrets controller
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml

# Encrypt secret
kubectl create secret generic accellens-secrets \
  --from-literal=db-password='<REPLACE_WITH_PASSWORD>' \
  --dry-run=client -o yaml | \
  kubeseal -o yaml > sealed-secret.yaml

FastAPI configuration (test):

# apps/services/config.py
secrets_provider = "sops"
sops_secrets_path = "./secrets/test.accellens.sops.json"

The file specified in sops_secrets_path must be pre-decrypted by SOPS (e.g., via sops -d ... > test.accellens.sops.json) and contain JSON/YAML structure with keys listed in secrets_remote_keys.

4.3 Pre-Prod and Prod Environments

AWS Secrets Manager:

# Create secret
aws secretsmanager create-secret \
  --name accellens/staging/db-credentials \
  --secret-string '{"username":"<USERNAME>","password":"<PASSWORD>"}'

# Get secret
aws secretsmanager get-secret-value \
  --secret-id accellens/staging/db-credentials

FastAPI configuration (prod/pre-prod):

# apps/services/config.py
secrets_provider = "aws"
aws_secrets_manager_secret_arn = "arn:aws:secretsmanager:eu-central-1:123456789012:secret:accellens/prod/runtime"
aws_secrets_manager_region = "eu-central-1"
secrets_remote_keys = "database_url,redis_url,rabbitmq_url,s3_secret_key,openai_api_key"

HashiCorp Vault (optional):

secrets_provider = "vault"
vault_addr = "https://vault.example.com"
vault_token = "<CICD or Kubernetes service account token>"
vault_secret_path = "kv/data/accellens/prod/runtime"  # KV v2 supported

Rotation Policies (for prod):

aws secretsmanager rotate-secret \
  --secret-id accellens/prod/db-credentials \
  --rotation-lambda-arn arn:aws:lambda:region:account:function:rotate-secret

4.4 Integration in Python Services

  • Module apps/services/utils/secrets_provider.py implements abstraction for loading secrets from AWS Secrets Manager, HashiCorp Vault, or SOPS.
  • apps/services/config.py automatically calls SecretsProvider on initialization if secrets_provider is not env. Allowed values: env, aws, vault, sops.
  • Keys listed in secrets_remote_keys are supported (default: database_url, redis_url, rabbitmq_url, s3_secret_key, openai_api_key, anthropic_api_key, elevenlabs_api_key). Keys must match by name (case insensitive).
  • When using SOPS, it is recommended to install PyYAML if the file is stored in YAML; JSON is supported out of the box.
  • Secrets loading errors are logged but do not block startup (fallback to .env values), meeting the "infrastructure must come up" requirement even if Secret Manager is temporarily unavailable.

4.5 TLS/mTLS for Python Services and Celery

  1. Create Kubernetes secret with certificates (server cert/key + root CA for client certificates):
kubectl create secret generic accellens-prod-internal-tls \
  --from-file=tls.crt=./certs/services.crt \
  --from-file=tls.key=./certs/services.key \
  --from-file=ca.crt=./certs/rootCA.crt
  1. Enable mounting via Helm (infra/k8s/environments/<env>/values.yaml):
services:
  tls:
    enabled: true
    secretName: accellens-prod-internal-tls
    mountPath: /etc/accellens/tls
  env:
    TLS_ENABLED: 'true'
    TLS_CERT_FILE: /etc/accellens/tls/tls.crt
    TLS_KEY_FILE: /etc/accellens/tls/tls.key
    TLS_CA_FILE: /etc/accellens/tls/ca.crt
    TLS_REQUIRE_CLIENT_CERT: 'true'
    CELERY_TLS_ENABLED: 'true'
    CELERY_TLS_CA_FILE: /etc/accellens/tls/ca.crt
    CELERY_TLS_CERT_FILE: /etc/accellens/tls/tls.crt
    CELERY_TLS_KEY_FILE: /etc/accellens/tls/tls.key
celeryWorker:
  tls:
    enabled: true
    secretName: accellens-prod-internal-tls
  env:
    CELERY_TLS_ENABLED: 'true'
    CELERY_TLS_CA_FILE: /etc/accellens/tls/ca.crt
    CELERY_TLS_CERT_FILE: /etc/accellens/tls/tls.crt
    CELERY_TLS_KEY_FILE: /etc/accellens/tls/tls.key
  1. Environment Variables (see also docs/deployment/environment-variables.md):
Variable Purpose
TLS_ENABLED Enables HTTPS at FastAPI/uvicorn level
TLS_CERT_FILE Path to server certificate (PEM)
TLS_KEY_FILE Path to private key
TLS_CA_FILE Root CA for client cert validation
TLS_REQUIRE_CLIENT_CERT true for mTLS (TLS_CA_FILE required)
CELERY_TLS_ENABLED Enables TLS for RabbitMQ/Redis connections
CELERY_TLS_CA_FILE CA to verify broker
CELERY_TLS_CERT_FILE Client cert for RabbitMQ
CELERY_TLS_KEY_FILE Client key for RabbitMQ
  1. RabbitMQ/Redis addresses: use amqps:// and rediss:// schemes with issued certificates. Celery automatically applies TLS settings if CELERY_TLS_ENABLED=true is set.
  2. mTLS in FastAPI: apps/services/common.py enables client certificate verification if TLS_REQUIRE_CLIENT_CERT=true. TLS_CA_FILE must contain the CA that signed Gateway/Sidecar certificates.
  3. Observability: update prometheus/otel endpoints to https://services:8000 and ensure sidecar (Envoy/Istio) trusts the specified CA.

4.6 Secrets Rotation Checklist

  • AWS Secrets Manager (prod/pre-prod):

    1. Create new secret version (aws secretsmanager put-secret-value ...).
    2. Update secrets_remote_keys if new fields added.
    3. Run scripts/run-security-tests.sh (verify secrets are picked up).
    4. Update Helm values (aws_secrets_manager_secret_arn) and run helm upgrade.
    5. Log rotation in runbook (time, owner, ticket link).

  • Vault (QA/dev clusters):

    1. Issue new kv record vault kv put.
    2. Regenerate access token if expiring.
    3. Run smoke (scripts/run-security-tests.sh) + helm upgrade.
    4. Update team access (Vault policies).

  • SOPS (test/local):

    1. Decrypt file, update values, encrypt back sops -e.
    2. Recreate sealed secret/secret (kubectl apply -f ...).
    3. Save file hash in CHANGELOG (integrity check).

  • General Rules:

    • Keep rotation log (date, author, what changed).
    • Revoke old credentials (IAM, DB, RabbitMQ) after confirming successful deploy.
    • Automatically notify SRE/DevOps (Slack + ticket) about successful rotation completion.

4.7 Password Transmission Security

Critically Important: Passwords are sent in clear text in JSON payload during login. This is standard practice, but only when using HTTPS.

4.7.1 Security Requirements

  1. Production Environment:

    • HTTPS Mandatory for all authentication endpoints (/api/v1/auth/login, /api/v1/auth/register)
    • Gateway automatically blocks HTTP requests to auth endpoints in production
    • Use reverse proxy (nginx/traefik) or Kubernetes Ingress with TLS for SSL termination

  2. Development Environment:

    • HTTP allowed only for localhost and 127.0.0.1
    • For remote access use HTTPS (e.g., via ngrok or local reverse proxy)

  3. What HTTPS protects:

    • Traffic encryption between client and server
    • Protection against password interception (man-in-the-middle attacks)
    • Protection against request modification

4.7.2 HTTPS Check

Gateway automatically checks connection security:

// Checks:
// 1. request.protocol === 'https'
// 2. X-Forwarded-Proto header === 'https' (for reverse proxy)
// 3. X-Forwarded-Ssl header === 'on' (alternative method)

In Production: If HTTPS is not detected, request is blocked with error 403:

{
  "error": "Forbidden",
  "message": "HTTPS is required for authentication endpoints in production. Please use HTTPS to protect your credentials."
}

4.7.3 HTTPS Setup in Kubernetes

Ingress with TLS (recommended):

# infra/k8s/environments/prod/values.yaml
ingress:
  enabled: true
  className: nginx
  tls:
    - secretName: accellens-tls
      hosts:
        - api.accellens.dev
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'

Verify after deploy:

# Check if HTTPS works
curl -I https://api.accellens.dev/api/v1/auth/login

# Check certificate
openssl s_client -connect api.accellens.dev:443 -servername api.accellens.dev

4.7.4 HTTPS Setup for Local Development

Option 1: ngrok (for testing with remote devices):

ngrok http 8000
# Use https://xxxx.ngrok.io for access

Option 2: Local reverse proxy with self-signed certificate:

# nginx.conf
server {
    listen 8443 ssl;
    server_name localhost;

    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;

    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header X-Forwarded-Proto https;
    }
}

4.7.5 Additional Security Measures

  1. Rate Limiting: Configured (5 requests per minute on /api/v1/auth/login)
  2. Password Hashing: Passwords never stored in clear text on server (bcrypt)
  3. Audit Logging: All login attempts logged (success and failure)
  4. PII Redaction: Passwords automatically removed from logs

4.7.6 Security Checklist

  • HTTPS used for all auth endpoints in production
  • Kubernetes Ingress configured with TLS certificates
  • Verified that HTTP requests to auth endpoints are blocked in production
  • Certificates renew automatically (Let's Encrypt)
  • Rate limiting active
  • Audit logging enabled

5. Audit Logging

5.1 Dev Environment

Basic logging:

  • Logging of all user actions
  • Storage in PostgreSQL (optional)
  • Local logs to files

5.2 Test Environment

Extended logging:

  • All user actions
  • Security events (auth failures, rate limits)
  • API access logs
  • Storage in PostgreSQL + centralized logs

5.3 Pre-Prod Environment

Production-like audit logging:

  • All user actions
  • Security events with full details
  • API access logs with IP addresses
  • PII redaction in logs
  • Storage in PostgreSQL + centralized logs

5.4 Production Environment

Full audit logging with SIEM integration:

  • All user actions
  • Security events with escalation
  • API access logs
  • PII redaction mandatory
  • Storage in PostgreSQL + centralized logs + SIEM
  • Retention: minimum 1 year for compliance

SIEM Integration:

# SIEM export configuration
SIEM_ENABLED=true
SIEM_ENDPOINT=https://siem.example.com/api/logs
SIEM_API_KEY=<secret>

6. Security Scanning in CI

6.1 Static Application Security Testing (SAST)

Python (Bandit):

- name: Run Bandit
  run: bandit -r . -f json -o bandit-report.json

Quick run of local pipeline: ./scripts/run-security-tests.sh — runs Bandit with high severity + pytest-marker security.

TypeScript/JavaScript (ESLint security plugins):

  • eslint-plugin-security for detecting vulnerabilities
  • Automatically runs in lint workflow

6.2 Dependency Scanning

Dependabot:

  • Automatic PRs for dependency updates
  • Configured in .github/dependabot.yml

Trivy:

  • Python and Node.js dependency scanning
  • Known vulnerability detection (CVE)
  • GitHub Security integration

6.3 Docker Image Scanning

Trivy for Dockerfiles:

  • Docker configuration scanning
  • Vulnerability detection in base images
  • Best practices check

6.4 Infrastructure Scanning (Terraform)

Checkov:

  • Terraform configuration scanning
  • Security misconfiguration detection
  • Compliance check (CIS, AWS Security Best Practices)

6.5 Security audit process (monthly/quarterly)

Regular security audit process (triage monthly + deep dive quarterly) is described separately:

  • docs/deployment/security-audit-process.md

7. Swagger UI Security

7.1 Production Configuration

Mandatory Settings:

  1. Strong Password:
# Generate strong password (32+ chars)
openssl rand -base64 32

# Set in environment
export SWAGGER_AUTH_PASSWORD="<generated-password>"
  1. Change Username:
# Do not use 'admin'
export SWAGGER_AUTH_USERNAME="accellens-api-docs"
  1. Rate Limiting:
  • Automatically configured: 10 requests per minute
  • Brute force protection
  • Logging of all attempts
  1. Audit Logging:
  • All access attempts logged
  • Auth failures tracked
  • Rate limit violations logged

7.2 IP Whitelist (Infrastructure Level)

Nginx/Cloudflare:

location /api/docs {
    allow 192.168.1.0/24;  # Internal network
    allow 10.0.0.0/8;      # VPN network
    deny all;

    # Basic Auth still required
    auth_basic "Swagger UI";
    auth_basic_user_file /etc/nginx/.htpasswd;
}

Cloudflare Access Rules:

  • Configure IP whitelist in Cloudflare dashboard
  • Apply to /api/docs/* routes

7.3 Suspicious Activity Monitoring

Prometheus Alerts:

- alert: SwaggerUIBruteForce
  expr: rate(swagger_auth_failures_total[5m]) > 5
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: 'Multiple failed Swagger UI access attempts'

Audit Log Queries:

-- Find multiple failed attempts
SELECT ip_address, COUNT(*) as failures
FROM audit_logs
WHERE event_type = 'auth_failure'
  AND metadata->>'resource' = 'swagger_ui'
  AND created_at > NOW() - INTERVAL '1 hour'
GROUP BY ip_address
HAVING COUNT(*) > 10;

7.4 Password Rotation

Procedure:

  1. Generate new password
  2. Update in AWS Secrets Manager / Vault
  3. Update in Kubernetes secrets
  4. Restart gateway pods
  5. Notify team about new password (via secure channel)

Automation:

# Password rotation script
./scripts/rotate-swagger-password.sh

8. Best Practices

8.1 Secrets Management

  1. Never commit secrets to git
  2. Use secrets managers for all environments except dev
  3. Rotate secrets regularly (minimum quarterly for prod)
  4. Least privilege access to secrets

8.2 PII Handling

  1. Redaction before logging all user data
  2. Redaction before sending to LLM all prompts
  3. Audit logging of all PII detection instances
  4. Compliance with GDPR, HIPAA (where applicable)

8.3 Security Scanning

  1. Regular scanning (weekly minimum)
  2. Automatic updates via Dependabot
  3. Rapid fix of critical vulnerabilities
  4. Documentation of all security exceptions

8.3.1 Deprecation Warnings vs Security Vulnerabilities

Important: There is a critical difference between deprecation warnings and security vulnerabilities:

Deprecation Warnings (Informational):

  • Package is marked as deprecated and will be removed in future
  • NOT a security vulnerability
  • May indicate package is no longer supported
  • Requires migration planning, but not critical

Security Vulnerabilities (Critical):

  • Real security threats that can be exploited
  • Require immediate fix
  • Can lead to system compromise

Security Check:

# Check security (distinguishes deprecation and vulnerabilities)
npm run security:check

# Strict check (fails on critical vulnerabilities)
npm run security:check:strict

# Standard npm audit check
npm audit

# Auto-fix known vulnerabilities
npm audit fix

# Update packages with overrides
npm run update

Automatic Check in CI/CD:

  • Workflow .github/workflows/security.yml automatically checks security
  • Critical and high vulnerabilities block deploy (if configured)
  • Deprecation warnings are shown as info messages

Results Handling:

  1. Security Vulnerabilities (Critical/High):

    • Fix immediately via npm audit fix or update overrides in package.json
    • If auto-fix impossible, replace package with alternative
    • Document in security exceptions if fix impossible

  2. Deprecation Warnings:

    • Plan migration to supported alternative
    • Does not block deploy, but requires attention
    • Track via Dependabot updates

Known Issues and Solutions:

The project uses npm overrides to automatically fix the following vulnerabilities:

  • uglify-js → forced version 3.19.3
  • underscore → forced version 1.13.7
  • xmldom → replaced with @xmldom/xmldom@0.8.11

These fixes are applied automatically via overrides in package.json during npm install.

Important: Project uses npm 10.9.4 (constraint in package.json: "npm": ">=10.0.0 <11.0.0"). Issue with applying overrides to deep dependencies detected in npm 11.x.

8.4 Audit Logging

  1. Immutable logs for production
  2. Retention policies according to compliance requirements
  3. Regular review of audit logs for suspicious activity
  4. SIEM integration for production

9. Compliance

9.1 GDPR

  • PII redaction mandatory
  • Right to deletion
  • Data retention policies
  • Audit logging of all PII accesses

9.2 HIPAA (for healthcare clients)

  • Enhanced PII detection
  • Encrypted storage
  • Access controls
  • Audit trails

9.3 SOC2 (roadmap)

  • Security controls
  • Access management
  • Monitoring and alerting
  • Incident response procedures

10. Troubleshooting

10.1 Safety filters not working

Check:

  • ai_enable_safety_filters=true in config
  • Presidio installed (pip install presidio-analyzer presidio-anonymizer)
  • OpenAI API key configured (for moderation API)

10.2 PII not detected

Check:

  • Presidio Analyzer initialized
  • Logs show initialization errors
  • Text is in English (Presidio works best with English)

10.3 Audit logs not saved

Check:

  • PostgreSQL available
  • AUDIT_LOG_USE_POSTGRESQL=true
  • DB access permissions
  • Error logs in gateway

11. Roadmap

  • Enhanced PII detection for other languages
  • Automated security scanning in pre-commit hooks
  • Security dashboard in Grafana
  • Integration with security information systems
  • Automated incident response