DevSecOps Application with HashiCorp Vault

Table of Contents

1. Introduction
2. What is HashiCorp Vault?
3. Project Overview
4. Architecture
5. Prerequisites
6. Installation
7. Configuration
8. Usage
9. Security Features
10. CI/CD Pipeline
11. Development Workflow
12. Testing
13. Deployment
14. Troubleshooting
15. Contributing
16. License

---

Introduction

This project demonstrates a production-grade DevSecOps implementation using HashiCorp Vault as the centralized secrets management solution. It showcases modern software engineering practices including infrastructure as code, automated security scanning, continuous integration/continuous deployment (CI/CD), and secrets management best practices.

The application is built with Python Flask and integrates seamlessly with Vault to retrieve sensitive configuration data without exposing credentials in source code, environment variables, or configuration files.

---

What is HashiCorp Vault?

Overview

HashiCorp Vault is an identity-based secrets and encryption management system. It provides a unified interface to any secret while providing tight access control and recording a detailed audit log. Modern applications require access to secrets such as database credentials, API keys for external services, credentials for service-oriented architecture communication, and more. Vault provides these secrets in a secure, auditable manner.

Key Capabilities

Secrets Management Vault securely stores and tightly controls access to tokens, passwords, certificates, API keys, and other secrets. It provides a centralized workflow for distributing secrets across applications and systems, ensuring that sensitive data never appears in plain text in code repositories or configuration files.

Dynamic Secrets Vault can generate secrets on-demand for some systems, such as AWS, SQL databases, or other services. When an application needs to access a database, it asks Vault for credentials, and Vault generates a unique set of credentials with a specific time-to-live (TTL). Once the TTL expires, the credentials are automatically revoked.

Data Encryption Vault provides encryption as a service with centralized key management. Applications can encrypt and decrypt data without having to manage encryption keys themselves. Vault handles the complexity of key rotation and ensures that encryption keys are never exposed to applications.

Identity-Based Access Vault can authenticate and authorize users and applications based on trusted identities. It supports multiple authentication methods including tokens, username/password, LDAP, Kubernetes, cloud provider IAM, and more. Fine-grained policies control what secrets each identity can access.

Audit Logging Every interaction with Vault is logged to one or more audit devices. Audit logs record all requests and responses, including authentication attempts, secret access, policy changes, and administrative operations. This provides a complete audit trail for compliance and security analysis.

Why Use Vault?

Security by Default Traditional methods of storing secrets such as plaintext files, environment variables, or encrypted files in version control are fundamentally insecure. Vault ensures secrets are encrypted at rest and in transit, with fine-grained access controls and comprehensive audit logging.

Dynamic Credentials Static credentials pose significant security risks. If credentials are compromised, they remain valid until manually rotated. Vault's dynamic secrets are short-lived and automatically revoked, significantly reducing the attack surface.

Centralized Management Managing secrets across multiple applications, environments, and teams is complex and error-prone. Vault provides a single source of truth for all secrets, with a unified API and consistent access patterns.

Compliance and Governance Many regulatory frameworks require detailed audit trails of who accessed what data and when. Vault's comprehensive audit logging provides the evidence needed for compliance with standards such as SOC 2, PCI DSS, HIPAA, and GDPR.

---

Project Overview

Purpose

This project serves as a reference implementation for building secure, production-ready applications using modern DevSecOps practices. It demonstrates how to:

• Integrate HashiCorp Vault for secrets management
• Implement automated security scanning in CI/CD pipelines
• Build containerized applications following security best practices
• Manage infrastructure as code
• Implement comprehensive testing and quality gates
• Deploy applications securely with minimal manual intervention

Technology Stack

Application Layer

• Python 3.11: Modern, secure Python runtime
• Flask 3.0: Lightweight web framework for building RESTful APIs
• HVAC: Official HashiCorp Vault client library for Python
• Gunicorn: Production-grade WSGI HTTP server

Infrastructure Layer

• Docker: Container runtime for application isolation
• Docker Compose: Multi-container orchestration for local development
• HashiCorp Vault 1.15: Secrets management and encryption service

Security Tools

• Trivy: Comprehensive vulnerability scanner for containers and dependencies
• Bandit: Security linter specifically designed for Python code
• Gitleaks: Tool for detecting hardcoded secrets in source code
• SonarCloud: Continuous code quality and security analysis platform

Development Tools

• Pytest: Modern testing framework with extensive plugin ecosystem
• Pylint: Static code analyzer for Python
• Black: Opinionated code formatter for Python
• Make: Build automation tool for common development tasks

Key Features

Secure Secrets Management All sensitive data including database credentials, API keys, and encryption keys are stored in Vault. The application retrieves secrets at runtime using authenticated API calls. No secrets are ever committed to version control or stored in configuration files.

Automated Security Scanning The CI/CD pipeline includes multiple security checks that run automatically on every commit. Trivy scans for known vulnerabilities in dependencies and container images. Bandit analyzes Python code for common security issues. Gitleaks prevents accidental exposure of secrets in source code.

Container Security Hardening Application containers run as non-root users with minimal privileges. Images are based on slim base images to reduce attack surface. Health checks ensure containers are functioning correctly before accepting traffic.

Comprehensive Testing Unit tests validate business logic and integration with Vault. Code coverage tracking ensures adequate test coverage. Tests run automatically in CI/CD pipelines before deployment.

Infrastructure as Code All infrastructure is defined in version-controlled configuration files. Docker Compose defines the application stack for local development. Dockerfiles specify how to build application images. This ensures consistency across development, testing, and production environments.

Developer Experience A comprehensive Makefile provides simple commands for common tasks. Color-coded output improves readability. Documentation is extensive and kept up-to-date. The project structure follows best practices and is easy to navigate.

---

Architecture

System Architecture

The application follows a microservices-inspired architecture with clear separation of concerns:

Application Container The Flask application runs in its own container, isolated from other services. It communicates with Vault over HTTP to retrieve secrets. The application exposes a RESTful API for client interactions. Health check endpoints allow monitoring systems to verify application status.

Vault Container Vault runs in development mode for local testing, with data stored in memory. In production, Vault would use a persistent storage backend such as Consul or integrated storage. The Vault API is exposed on port 8200 for both the application and administrators.

Network Architecture Containers communicate over a Docker bridge network. The application resolves Vault by service name using Docker's built-in DNS. Only necessary ports are exposed to the host system. In production, network policies would further restrict traffic between services.

Security Architecture

Authentication Flow The application authenticates to Vault using a token during initialization. In production, AppRole or Kubernetes authentication would be used instead of static tokens. Vault verifies the token and returns an authenticated session.

Authorization Model Vault policies define what secrets each identity can access. The application policy grants read-only access to secrets under the app/ path. Vault denies access to any secrets not explicitly allowed by policy.

Secret Retrieval Flow When the application needs a secret, it makes an authenticated API call to Vault. Vault verifies the token has permission to access the requested secret. If authorized, Vault returns the secret value. The application uses the secret and never logs or persists it.

Audit Trail Every Vault interaction is logged to audit devices. Logs include timestamp, requesting identity, requested path, and result. Audit logs are immutable and suitable for compliance requirements.

Data Flow

1. Client sends HTTP request to application API endpoint
2. Application determines which secrets are needed to fulfill request
3. Application authenticates to Vault and requests secrets
4. Vault validates authentication token and checks authorization policies
5. Vault returns requested secrets if authorized, otherwise denies access
6. Application uses secrets to perform required operations
7. Application returns response to client
8. All Vault interactions are logged to audit devices

---

Prerequisites

Before installing this project, ensure you have the following software installed on your system:

Required Software

Docker Engine 20.10 or later

• Installation: https://docs.docker.com/engine/install/
• Verify installation: docker --version

Docker Compose 2.0 or later

• Installation: https://docs.docker.com/compose/install/
• Verify installation: docker-compose --version

Python 3.11 or later

• Installation: https://www.python.org/downloads/
• Verify installation: python3 --version

Make utility

• Linux/macOS: Usually pre-installed
• Windows: Install via chocolatey (choco install make) or use WSL2
• Verify installation: make --version

Optional Software

Git for version control

• Installation: https://git-scm.com/downloads
• Verify installation: git --version

jq for JSON formatting in terminal

• Installation: https://stedolan.github.io/jq/download/
• Verify installation: jq --version

HashiCorp Vault CLI for manual Vault operations

• Installation: https://www.vaultproject.io/downloads
• Verify installation: vault --version

System Requirements

Minimum:

• 2 CPU cores
• 4 GB RAM
• 10 GB free disk space

Recommended:

• 4 CPU cores
• 8 GB RAM
• 20 GB free disk space

Network Requirements

The following ports must be available on your system:

• Port 5000: Flask application
• Port 8200: Vault API and UI

---

Installation

Step 1: Clone the Repository

git clone https://github.com/yourusername/devsecops-vault-project.git
cd devsecops-vault-project

Step 2: Initial Setup

Run the setup command to create necessary configuration files:

make setup

This creates a .env file from the template. Review and modify settings as needed.

Step 3: Install Dependencies

Install Python dependencies for local development:

make install

This installs all required Python packages including Flask, HVAC, and development tools.

Step 4: Build Docker Images

Build the application Docker image:

make build

This builds the Docker image according to the Dockerfile specifications.

Step 5: Start Services

Start all services including Vault and the application:

make start

This command performs the following actions:

• Starts Vault in development mode
• Waits for Vault to become healthy
• Initializes Vault with sample secrets
• Starts the Flask application
• Verifies all services are running correctly

Step 6: Verify Installation

Check that all services are running:

make health

You should see healthy responses from both the application and Vault.

---

Configuration

Environment Variables

The application uses environment variables for configuration. Create a .env file based on .env.example:

Vault Configuration

VAULT_ADDR=http://localhost:8200

The URL where Vault API is accessible. In production, this should use HTTPS.

VAULT_TOKEN=dev-token

Authentication token for Vault. In development mode, the root token is dev-token. In production, use AppRole or Kubernetes authentication instead of static tokens.

VAULT_NAMESPACE=

Vault namespace if using Vault Enterprise. Leave empty for Vault OSS.

Application Configuration

DEBUG=False

Enable or disable debug mode. Never enable debug mode in production as it exposes sensitive information.

SECRET_KEY=change-me-in-production

Secret key for Flask session encryption. Generate a random value for production using python -c "import secrets; print(secrets.token_hex(32))".

Vault Configuration

Secrets Storage

Secrets are stored in Vault's KV v2 secrets engine at the following paths:

secret/app/database - Database connection credentials
secret/app/api - External API keys and endpoints

Access Policies

The application uses a restrictive policy that grants read-only access to application secrets:

path "secret/data/app/*" {
  capabilities = ["read", "list"]
}

Initial Secrets

Sample secrets are automatically created during make start:

Database credentials:

• host: postgresql.example.com
• port: 5432
• username: app_user
• password: secure_password_123

API configuration:

• key: api_key_xyz_123
• endpoint: https://api.example.com

Docker Configuration

docker-compose.yml

Defines the application stack including Vault and the Flask application. Modify this file to add additional services or change network configuration.

Dockerfile

Defines how to build the application container image. The image is based on Python 3.11 slim and includes security hardening measures such as running as a non-root user.

---

Usage

Starting the Application

Start all services:

make start

The application will be available at http://localhost:5000 and Vault UI at http://localhost:8200.

API Endpoints

Health Check

curl http://localhost:5000/health

Returns application health status. Used by monitoring systems and load balancers.

Retrieve Secret Example

curl http://localhost:5000/api/secret

Demonstrates retrieving database credentials from Vault. Returns connection information without exposing the password.

Configuration Example

curl http://localhost:5000/api/config

Demonstrates retrieving API configuration from Vault. Returns whether API credentials are configured.

Using the Vault UI

Access the Vault UI at http://localhost:8200 and login with token dev-token.

Navigate to the Secrets section to view and manage secrets. The UI provides a convenient way to:

• Browse existing secrets
• Create new secrets
• Update secret values
• View secret metadata and version history
• Manage access policies

Using the Vault CLI

List secrets:

make vault-secrets

Read database secret:

make vault-read-db

Read API secret:

make vault-read-api

Open Vault shell for manual operations:

make vault-shell

Viewing Logs

View all logs:

make logs

View application logs only:

make logs-app

View Vault logs only:

make logs-vault

Stopping Services

Stop all services:

make stop

Stop and remove all containers, networks, and volumes:

make destroy

---

Security Features

Secrets Management

No Hardcoded Secrets All secrets are stored in Vault and retrieved at runtime. Source code, Docker images, and configuration files contain no sensitive information.

Dynamic Secret Retrieval Secrets are fetched on-demand when needed. The application never caches secrets in memory longer than necessary.

Audit Logging Every secret access is logged in Vault's audit log with timestamp, identity, and requested path.

Vulnerability Scanning

Dependency Scanning Trivy scans Python dependencies for known CVEs. The CI/CD pipeline fails if high or critical vulnerabilities are detected.

Container Image Scanning Trivy scans Docker images for vulnerabilities in base images and installed packages.

Code Scanning Bandit analyzes Python code for security issues such as SQL injection, hardcoded passwords, and insecure functions.

Secret Detection

Pre-commit Protection Gitleaks scans commits for accidentally included secrets such as API keys, passwords, and tokens.

Repository Scanning Gitleaks scans the entire repository history to detect any secrets that may have been committed in the past.

Code Quality

Static Analysis Pylint analyzes code for bugs, code smells, and style violations. The CI/CD pipeline enforces a minimum quality threshold.

Code Coverage Pytest measures test coverage and generates reports. The pipeline requires a minimum coverage percentage before allowing deployment.

Code Formatting Black automatically formats code to ensure consistency. Isort organizes imports according to best practices.

Container Security

Non-Root User The application container runs as a non-privileged user, not as root. This limits the impact of potential container escapes.

Minimal Base Image The container uses a slim Python image to reduce attack surface. Only necessary packages are installed.

Health Checks Docker health checks ensure the application is functioning correctly. Unhealthy containers are automatically restarted.

Resource Limits Production deployments should specify CPU and memory limits to prevent resource exhaustion attacks.

---

CI/CD Pipeline

Pipeline Stages

The GitHub Actions workflow includes the following stages:

Security Scanning Stage

Runs multiple security scanners in parallel:

• Trivy scans filesystem for vulnerabilities
• Bandit scans Python code for security issues
• Gitleaks scans for hardcoded secrets
• Results are uploaded to GitHub Security tab

Code Quality Stage

Analyzes code quality:

• Pylint performs static code analysis
• Pytest runs unit tests with coverage measurement
• SonarCloud performs comprehensive quality analysis
• Pipeline fails if quality gates are not met

Build and Push Stage

Builds and publishes Docker images:

• Builds Docker image using BuildKit for efficiency
• Pushes image to GitHub Container Registry
• Tags image with both latest and commit SHA
• Scans built image for vulnerabilities

Deployment Stage

Deploys to target environment:

• Updates deployment manifests
• Applies infrastructure changes
• Performs smoke tests
• Rolls back on failure

Local CI Simulation

Run the complete CI pipeline locally before pushing:

make ci-local

This runs all checks that will run in CI:

• Code formatting verification
• Linting
• Unit tests
• Security scans

Fix any issues before pushing to avoid pipeline failures.

Pipeline Configuration

The pipeline is defined in .github/workflows/ci-cd.yml. Key configuration options:

Trigger Events

• Push to main or develop branches
• Pull requests to main branch

Secrets Required

• GITHUB_TOKEN: Automatically provided by GitHub
• SONAR_TOKEN: SonarCloud authentication token

Artifacts

• Test coverage reports
• Security scan results
• Built Docker images

---

Development Workflow

Daily Development Workflow

Start your development session:

make start

Make code changes in your editor. The application supports hot reloading during development.

Run tests after making changes:

make test

View application logs to debug issues:

make logs-app

Format code before committing:

make format

Run linting to catch issues:

make lint

Run local CI pipeline before pushing:

make ci-local

Stop services at end of session:

make stop

Adding New Secrets

To add a new secret to Vault:

1. Open Vault shell:

make vault-shell

2. Create the secret:

vault kv put secret/app/newsecret key=value

3. Update the Vault policy if needed to grant access

4. Update the application code to retrieve the secret:

secret = vault.get_secret('app/data/newsecret')

Modifying the Application

The application code is organized as follows:

• app/main.py: API endpoints and request handlers
• app/vault_client.py: Vault integration logic
• app/config.py: Configuration management
• tests/: Unit tests

After modifying code, rebuild and restart the application:

make app-rebuild

Working with Vault Policies

Vault policies are defined in HCL files in vault/policies/.

To update a policy:

1. Edit the policy file
2. Apply the updated policy:

vault policy write app-policy /vault/policies/app-policy.hcl

3. Restart the application to use the new policy

---

Testing

Running Tests

Run all tests:

make test

Run tests with coverage report:

make test-coverage

Run tests in watch mode (automatically re-run on file changes):

make test-watch

Writing Tests

Tests are located in the tests/ directory and use pytest.

Example test structure:

import unittest
from app.main import app

class TestAPI(unittest.TestCase):
    def setUp(self):
        self.client = app.test_client()
    
    def test_endpoint(self):
        response = self.client.get('/api/endpoint')
        self.assertEqual(response.status_code, 200)

Test Coverage

The project aims for high test coverage. Coverage reports show which lines of code are executed during tests.

View coverage report:

make test-coverage
open htmlcov/index.html

Integration Testing

Integration tests verify the application works correctly with Vault:

def test_vault_integration(self):
    secret = vault.get_secret('app/data/database')
    self.assertIsNotNone(secret)
    self.assertIn('host', secret)

---

Deployment

Production Considerations

Vault Configuration

For production deployments:

1. Use Vault in server mode, not development mode
2. Configure a persistent storage backend (Consul, Integrated Storage, etc.)
3. Enable TLS for all Vault communication
4. Use AppRole or Kubernetes authentication instead of tokens
5. Enable audit logging to a persistent location
6. Configure Vault for high availability with multiple instances
7. Implement automated secret rotation policies
8. Set up Vault monitoring and alerting

Application Configuration

For production deployments:

1. Use environment-specific configuration files
2. Enable application logging to a centralized logging system
3. Configure resource limits for containers
4. Implement horizontal pod autoscaling
5. Set up health checks and liveness probes
6. Configure network policies to restrict traffic
7. Enable application performance monitoring
8. Implement distributed tracing

Security Hardening

Additional security measures for production:

1. Scan images for vulnerabilities before deployment
2. Sign images to ensure authenticity
3. Use minimal base images
4. Run containers as non-root users
5. Enable read-only root filesystems where possible
6. Configure security contexts and pod security policies
7. Implement network segmentation
8. Enable audit logging at all layers

Deployment to Kubernetes

Example Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: devsecops-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: devsecops-app
  template:
    metadata:
      labels:
        app: devsecops-app
    spec:
      serviceAccountName: devsecops-app
      containers:
      - name: app
        image: ghcr.io/yourusername/devsecops-app:latest
        ports:
        - containerPort: 5000
        env:
        - name: VAULT_ADDR
          value: "https://vault.example.com"
        livenessProbe:
          httpGet:
            path: /health
            port: 5000
        readinessProbe:
          httpGet:
            path: /health
            port: 5000

Monitoring and Observability

Implement comprehensive monitoring:

Metrics

• Application request rate, latency, and error rate
• Vault request rate and latency
• Container resource usage (CPU, memory, network)
• Secret access patterns

Logging

• Application logs with structured logging
• Vault audit logs
• Container logs
• Security event logs

Alerting

• High error rates
• Slow response times
• Vault authentication failures
• Security scan failures
• Unusual secret access patterns

---

Troubleshooting

Common Issues

Vault Connection Refused

Symptom: Application fails to connect to Vault with "connection refused" error.

Solution:

# Check if Vault is running
make status

# Check Vault health
make vault-status

# View Vault logs
make logs-vault

# Restart Vault if needed
make restart

Vault Authentication Failed

Symptom: Application receives "permission denied" from Vault.

Solution:

# Verify token is correct
echo $VAULT_TOKEN

# Check token validity
vault token lookup

# Verify policy grants necessary permissions
vault policy read app-policy

# Re-initialize Vault if needed
make vault-init

Secret Not Found

Symptom: Application fails to retrieve secret from Vault.

Solution:

# List available secrets
make vault-secrets

# Verify secret exists
make vault-read-db

# Check secret path is correct in application code
# Ensure policy grants access to the path

Container Health Check Failing

Symptom: Container restarts repeatedly with health check failures.

Solution:

# View container logs
make logs-app

# Check application is starting correctly
docker exec devsecops-app ps aux

# Manually test health endpoint
curl http://localhost:5000/health

# Increase health check initial delay if needed

Port Already in Use

Symptom: Cannot start services due to port conflict.

Solution:

# Check what is using the port
lsof -i :5000
lsof -i :8200

# Stop conflicting service or change port in docker-compose.yml

Getting Help

If you encounter issues not covered here:

1. Check the application logs: make logs
2. Review Vault documentation: https://www.vaultproject.io/docs
3. Search existing GitHub issues
4. Open a new issue with detailed information:
   • Steps to reproduce the problem
   • Error messages and logs
   • Environment details (OS, Docker version, etc.)

---

Contributing

Contributions are welcome and appreciated. Please follow these guidelines:

Code Style

• Follow PEP 8 style guide for Python code
• Use Black for code formatting
• Use Isort for import sorting
• Write descriptive commit messages

Pull Request Process

1. Fork the repository
2. Create a feature branch: git checkout -b feature/your-feature
3. Make your changes
4. Run tests: make test
5. Run security scans: make security-scan
6. Format code: make format
7. Commit changes: git commit -m "Add feature"
8. Push to your fork: git push origin feature/your-feature
9. Open a pull request with detailed description

Code Review

All pull requests require review before merging. Reviewers will check:

• Code quality and style
• Test coverage
• Security implications
• Documentation updates
• Backward compatibility

---

License

This project is licensed under the MIT License. See the LICENSE file for details.

---

Acknowledgments

This project uses the following open source software:

• HashiCorp Vault for secrets management
• Flask web framework
• Docker containerization platform
• Trivy security scanner
• Bandit Python security linter
• GitHub Actions for CI/CD

---

Contact

For questions or support, please open an issue on GitHub or contact the maintainers:

• Project Repository: https://github.com/ines312692/devsecops-vault-project
• Issue Tracker: https://github.com/ines312692/devsecops-vault-project/issues
• Documentation: https://github.com/ines312692/devsecops-vault-project/wiki

---

Last Updated: October 2025