Skip to content

Feature: User-Defined Vault Implementations #101

@stack72

Description

@stack72

User-Defined Vault Implementations

Enable users to create custom vault implementations following the same pattern as extension models. Users will export a vault object from TypeScript files that defines the vault type, configuration schema, and provider factory.

Design Principles

  1. Minimal Dependencies: User vault code should not require external dependencies. The provider implementation can use any deps it needs
  2. Follow Extension Model Pattern: Use the same loading, validation, and registration patterns as UserModelLoader
  3. Type Safety: Use Zod schemas for configuration validation
  4. Backward Compatible: Existing built-in vaults work unchanged

Complete User Vault Example

// extensions/vaults/hashicorp.ts
import { z } from "npm:zod@4";

/**
 * VaultProvider interface that user vaults must implement.
 * This matches the interface from swamp's vault system.
 */
interface VaultProvider {
  get(secretKey: string): Promise<string>;
  put(secretKey: string, secretValue: string): Promise<void>;
  list(): Promise<string[]>;
  getName(): string;
}

/**
 * Configuration for HashiCorp Vault provider.
 */
interface HashiCorpConfig {
  /** Vault server address (e.g., "https://vault.example.com:8200") */
  address: string;
  /** Environment variable containing the Vault token (default: VAULT_TOKEN) */
  token_env?: string;
  /** Vault namespace (for enterprise) */
  namespace?: string;
  /** KV secrets engine mount path (default: "secret") */
  mount_path?: string;
  /** Path prefix for secrets (default: "swamp") */
  path_prefix?: string;
}

/**
 * HashiCorp Vault provider implementation.
 * Uses the HTTP API to store and retrieve secrets from HashiCorp Vault.
 */
class HashiCorpVaultProvider implements VaultProvider {
  private readonly name: string;
  private readonly address: string;
  private readonly tokenEnv: string;
  private readonly namespace?: string;
  private readonly mountPath: string;
  private readonly pathPrefix: string;

  constructor(name: string, config: Record<string, unknown>) {
    const typedConfig = config as HashiCorpConfig;
    
    this.name = name;
    this.address = typedConfig.address.replace(/\/$/, ""); // Remove trailing slash
    this.tokenEnv = typedConfig.token_env ?? "VAULT_TOKEN";
    this.namespace = typedConfig.namespace;
    this.mountPath = typedConfig.mount_path ?? "secret";
    this.pathPrefix = typedConfig.path_prefix ?? "swamp";
  }

  getName(): string {
    return this.name;
  }

  /**
   * Retrieves a secret from HashiCorp Vault.
   */
  async get(secretKey: string): Promise<string> {
    const token = this.getToken();
    const url = this.buildUrl(secretKey);

    const response = await fetch(url, {
      method: "GET",
      headers: this.buildHeaders(token),
    });

    if (!response.ok) {
      if (response.status === 404) {
        throw new Error(
          `Secret '${secretKey}' not found in HashiCorp Vault '${this.name}'`
        );
      }
      const error = await response.text();
      throw new Error(
        `Failed to retrieve secret '${secretKey}' from HashiCorp Vault: ${error}`
      );
    }

    const data = await response.json();
    
    // KV v2 returns data nested under data.data
    const secretData = data.data?.data ?? data.data;
    
    if (!secretData?.value) {
      throw new Error(
        `Secret '${secretKey}' exists but has no 'value' field in HashiCorp Vault '${this.name}'`
      );
    }

    return secretData.value;
  }

  /**
   * Stores a secret in HashiCorp Vault.
   */
  async put(secretKey: string, secretValue: string): Promise<void> {
    const token = this.getToken();
    const url = this.buildUrl(secretKey);

    const response = await fetch(url, {
      method: "POST",
      headers: this.buildHeaders(token),
      body: JSON.stringify({
        data: { value: secretValue },
      }),
    });

    if (!response.ok) {
      const error = await response.text();
      throw new Error(
        `Failed to store secret '${secretKey}' in HashiCorp Vault: ${error}`
      );
    }
  }

  /**
   * Lists all secret keys in the configured path.
   */
  async list(): Promise<string[]> {
    const token = this.getToken();
    // LIST endpoint uses metadata path for KV v2
    const url = `${this.address}/v1/${this.mountPath}/metadata/${this.pathPrefix}?list=true`;

    const response = await fetch(url, {
      method: "GET", // Vault accepts GET with ?list=true or LIST method
      headers: this.buildHeaders(token),
    });

    if (!response.ok) {
      if (response.status === 404) {
        // No secrets yet - return empty list
        return [];
      }
      const error = await response.text();
      throw new Error(
        `Failed to list secrets in HashiCorp Vault '${this.name}': ${error}`
      );
    }

    const data = await response.json();
    const keys: string[] = data.data?.keys ?? [];
    
    // Filter out directory entries (ending with /)
    return keys.filter((k: string) => !k.endsWith("/")).sort();
  }

  /**
   * Gets the Vault token from the configured environment variable.
   */
  private getToken(): string {
    const token = Deno.env.get(this.tokenEnv);
    if (!token) {
      throw new Error(
        `HashiCorp Vault token not found. Set the ${this.tokenEnv} environment variable.`
      );
    }
    return token;
  }

  /**
   * Builds the API URL for a secret key.
   * Uses KV v2 data path format.
   */
  private buildUrl(secretKey: string): string {
    return `${this.address}/v1/${this.mountPath}/data/${this.pathPrefix}/${secretKey}`;
  }

  /**
   * Builds HTTP headers for Vault API requests.
   */
  private buildHeaders(token: string): Record<string, string> {
    const headers: Record<string, string> = {
      "X-Vault-Token": token,
      "Content-Type": "application/json",
    };

    if (this.namespace) {
      headers["X-Vault-Namespace"] = this.namespace;
    }

    return headers;
  }
}

/**
 * User vault export - this is what swamp loads and validates.
 */
export const vault = {
  // Type identifier (used in CLI: swamp vault create hashicorp my-vault)
  type: "hashicorp",

  // Display name shown in CLI help and listings
  name: "HashiCorp Vault",

  // Description shown in CLI help
  description:
    "Store secrets in HashiCorp Vault using the HTTP API. " +
    "Requires VAULT_TOKEN environment variable (or custom token_env config). " +
    "Supports KV v2 secrets engine.",

  // Zod schema for validating provider configuration
  configSchema: z.object({
    address: z.string().url().describe("Vault server address"),
    token_env: z.string().optional().describe("Env var containing Vault token"),
    namespace: z.string().optional().describe("Vault namespace (enterprise)"),
    mount_path: z.string().optional().describe("KV secrets engine mount path"),
    path_prefix: z.string().optional().describe("Path prefix for secrets"),
  }),

  // Factory function called by VaultService to create the provider
  createProvider(name: string, config: Record<string, unknown>): VaultProvider {
    return new HashiCorpVaultProvider(name, config);
  },
};

Usage

Once the user vault is in place, use it like any built-in vault:

# Create a vault instance using the hashicorp type
swamp vault create hashicorp my-hcv --config '{"address": "https://vault.example.com:8200"}'

# Store a secret
swamp vault put my-hcv api-key "sk-12345"

# Retrieve a secret (in workflows via vault expressions)
# ${{ vault.my-hcv.api-key }}

# List secret keys
swamp vault list-keys my-hcv

Automatic Loading (Same as Extension Models)

User vaults will be automatically loaded at CLI startup using the same pattern as loadUserModels() in src/cli/mod.ts:

// New function in src/cli/mod.ts (mirrors loadUserModels)
async function loadUserVaults(): Promise<void> {
  const cwd = Deno.cwd();
  const markerRepo = new RepoMarkerRepository();

  try {
    const repoPath = RepoPath.create(cwd);
    const marker = await markerRepo.read(repoPath);

    const vaultsDir = resolveVaultsDir(marker);
    const absoluteVaultsDir = isAbsolute(vaultsDir)
      ? vaultsDir
      : resolve(cwd, vaultsDir);

    const loader = new UserVaultLoader();
    const result = await loader.loadVaults(absoluteVaultsDir);

    // Log failures as warnings (don't block CLI startup)
    for (const failure of result.failed) {
      console.error(
        `Warning: Failed to load user vault ${failure.file}: ${failure.error}`,
      );
    }
  } catch (error) {
    // Not in a swamp repo or other error - log at debug level
    if (Deno.env.get("SWAMP_DEBUG")) {
      console.debug(`Skipping user vaults: ${error}`);
    }
  }
}

export function resolveVaultsDir(marker: RepoMarkerData | null): string {
  // Environment variable takes highest priority
  const envVaultsDir = Deno.env.get("SWAMP_VAULTS_DIR");
  if (envVaultsDir) {
    return envVaultsDir;
  }

  // Then .swamp.yaml config
  if (marker?.vaultsDir) {
    return marker.vaultsDir;
  }

  // Default
  return "extensions/vaults";
}

export async function runCli(args: string[]): Promise<void> {
  // Load user models before setting up CLI
  await loadUserModels();
  // Load user vaults before setting up CLI
  await loadUserVaults();
  
  const cli = new Command()
    // ... rest of CLI setup
}

Implementation Plan

1. Create VaultTypeRegistry (src/domain/vaults/vault_type_registry.ts)

Replace static VAULT_TYPES array with a dynamic registry:

export interface VaultTypeInfo {
  type: string;
  name: string;
  description: string;
  configSchema?: z.ZodTypeAny;
  createProvider?: (name: string, config: Record<string, unknown>) => VaultProvider;
  isBuiltIn: boolean;
}

export class VaultTypeRegistry {
  private types = new Map<string, VaultTypeInfo>();

  register(info: VaultTypeInfo): void;
  get(type: string): VaultTypeInfo | undefined;
  getAll(): VaultTypeInfo[];
  has(type: string): boolean;
}

export const vaultTypeRegistry = new VaultTypeRegistry();

2. Create UserVaultLoader (src/domain/vaults/user_vault_loader.ts)

Mirror the UserModelLoader pattern exactly:

  • Discover .ts files in user vaults directory (excluding *_test.ts)
  • Dynamically import each file using import(\file://${absolutePath}`)`
  • Validate vault export against UserVaultSchema
  • Register with vaultTypeRegistry
const UserVaultSchema = z.object({
  type: z.string().regex(/^[a-z][a-z0-9_-]*$/),
  name: z.string(),
  description: z.string(),
  configSchema: z.custom<z.ZodTypeAny>((val) => val instanceof z.ZodType),
  createProvider: z.custom<CreateProviderFn>((val) => typeof val === "function"),
});

export class UserVaultLoader {
  async loadVaults(vaultsDir: string): Promise<LoadResult> {
    const result: LoadResult = { loaded: [], failed: [] };

    // Check if directory exists
    try {
      await Deno.stat(vaultsDir);
    } catch {
      return result; // No user vaults directory - not an error
    }

    const files = await this.discoverVaults(vaultsDir);

    for (const file of files) {
      try {
        const absolutePath = resolve(vaultsDir, file);
        const module = await import(`file://${absolutePath}`);

        if (!module.vault) {
          result.failed.push({ file, error: "No 'vault' export found" });
          continue;
        }

        // Validate the vault structure
        const parsed = UserVaultSchema.safeParse(module.vault);
        if (!parsed.success) {
          result.failed.push({ file, error: parsed.error.message });
          continue;
        }

        // Register with vault type registry
        const userVault = parsed.data;
        if (!vaultTypeRegistry.has(userVault.type)) {
          vaultTypeRegistry.register({
            type: userVault.type,
            name: userVault.name,
            description: userVault.description,
            configSchema: userVault.configSchema,
            createProvider: userVault.createProvider,
            isBuiltIn: false,
          });
          result.loaded.push(file);
        } else {
          result.failed.push({
            file,
            error: `Vault type '${userVault.type}' already registered`,
          });
        }
      } catch (error) {
        result.failed.push({ file, error: String(error) });
      }
    }

    return result;
  }

  private async discoverVaults(dir: string): Promise<string[]> {
    const files: string[] = [];
    for await (const entry of Deno.readDir(dir)) {
      if (
        entry.isFile && entry.name.endsWith(".ts") &&
        !entry.name.endsWith("_test.ts")
      ) {
        files.push(entry.name);
      }
    }
    return files.sort();
  }
}

3. Modify VaultService (src/domain/vaults/vault_service.ts)

Update registerVault() to use the registry instead of hardcoded switch:

registerVault(config: VaultConfiguration): void {
  const typeInfo = vaultTypeRegistry.get(config.type);

  if (!typeInfo) {
    const available = vaultTypeRegistry.getAll().map(t => t.type).join(", ");
    throw new Error(`Unknown vault type: ${config.type}. Available: ${available}`);
  }

  // Validate config against schema if present
  if (typeInfo.configSchema) {
    typeInfo.configSchema.parse(config.config);
  }

  // Create provider using factory or built-in logic
  let provider: VaultProvider;
  if (typeInfo.createProvider) {
    provider = typeInfo.createProvider(config.name, config.config);
  } else {
    // Fall back to built-in provider creation for backward compat
    provider = this.createBuiltInProvider(config);
  }

  this.providers.set(config.name, provider);
}

4. Update vault_types.ts

Convert to register built-in types with the new registry:

// Register built-in vault types
vaultTypeRegistry.register({
  type: "aws",
  name: "AWS Secrets Manager",
  description: "...",
  isBuiltIn: true,
});

vaultTypeRegistry.register({
  type: "local_encryption",
  name: "Local Encryption",
  description: "...",
  isBuiltIn: true,
});

// Export helpers for backward compatibility
export function getVaultTypes(): VaultTypeInfo[] {
  return vaultTypeRegistry.getAll();
}

export function getVaultType(type: string): VaultTypeInfo | undefined {
  return vaultTypeRegistry.get(type);
}

Files to Create

File Purpose
src/domain/vaults/vault_type_registry.ts Dynamic vault type registry
src/domain/vaults/user_vault_loader.ts User vault discovery and loading
src/domain/vaults/user_vault_loader_test.ts Tests for loader
src/domain/vaults/vault_type_registry_test.ts Tests for registry

Files to Modify

File Changes
src/domain/vaults/vault_types.ts Register built-ins with registry
src/domain/vaults/vault_service.ts Use registry for provider creation
src/domain/vaults/vaults.ts Export new modules
src/cli/commands/vault_create.ts Support user vault types
src/cli/mod.ts Add resolveVaultsDir(), loadUserVaults(), call in runCli()
src/infrastructure/persistence/repo_marker_repository.ts Add vaultsDir to RepoMarkerData

User Vault Location

User vaults will be stored in extensions/vaults/ directory (matching extensions/models/ for user models).

Resolution priority (same as models):

  1. SWAMP_VAULTS_DIR environment variable
  2. .swamp.yaml config (vaultsDir field)
  3. Default extensions/vaults

Key Requirements

  1. Zod Version: User vaults must use import { z } from "npm:zod@4"; (same as extension models)
  2. Export Name: Must export vault (not default)
  3. File Location: extensions/vaults/*.ts (excluding *_test.ts)
  4. Type Naming: Use lowercase with hyphens (e.g., hashicorp, azure-keyvault)

Verification

  1. deno check - Type checking passes
  2. deno lint - No lint errors
  3. deno fmt - Code formatted
  4. deno run test - All tests pass
  5. Manual testing:
    • Create a simple user vault (e.g., env vault that reads from env vars)
    • Run swamp vault create env my-env-vault
    • Run swamp vault put my-env-vault test-key test-value
    • Run swamp vault list-keys my-env-vault
    • Verify user vault appears in swamp vault types output

Metadata

Metadata

Assignees

No one assigned

    Labels

    betaIssues required to close out before public betaenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions