[forge] Keep working on the framework

docs/orchestration/vaults.md

@@ -0,0 +1,291 @@
+# Vaults and Secrets Management
+
+FORGE provides a secure vault system for managing sensitive
+information across projects. This system ensures that secrets are
+properly organized, documented, and validated.
+
+## Overview
+
+The vault system manages:
+- API keys and tokens
+- Cluster credentials
+- Database passwords
+- Service account keys
+- Certificate files
+- Configuration secrets
+
+## Vault Architecture
+
+### Vault Definitions
+
+Vaults are defined in YAML files located in `$FORGE_HOME/vaults/*.yaml`. Each vault specifies:
+
+- **env_key**: Environment variable pointing to the vault's secret directory
+- **description**: Documentation of the vault's purpose
+- **content**: List of files/secrets within the vault
+
+Example vault definition (`/vaults/psap-forge-ci.yaml`):
+
+```yaml
+env_key: PSAP_FORGE_FOURNOS_CI_SECRET_PATH
+description: |
+  Vault to access FOURNOS Kube API in the PSAP-mgmt cluster
+
+content:
+  kubeconfig:
+    description: |
+      KUBECONFIG file to access the PSAP-mgmt cluster with limited privileges
+
+  server_url:
+    description: |
+      URL of the Kube-API server (having it in a dedicate file allows
+      the CI engine to censor it in any output).
+```
+
+### Project Vault Requirements
+
+Projects should validate the vault they use:
+
+```python
+from projects.core.library import vault
+
+vault.init(["vault-1", "vault-2"])
+```
+
+This validates that the vaults...
+1. exist (the env key exists and points to an existing directory)
+2. have the expected content (the files listed in the vault YAML file exist)
+3. do not contain un-referenced files
+
+Currently the content of the files isn't validated, and the content of
+the vault is not versioned. This could be added in the future.
+
+## Vault Content Management
+
+### Content Definition Formats
+
+**Standard format** (recommended):
+```yaml
+content:
+  kubeconfig:
+    file: kubeconfig  # optional, defaults to content name
+    description: |
+      KUBECONFIG file for cluster access
+
+  api_token:
+    file: api.token
+    description: |
+      Authentication token for API access
+```
+
+### File Path Resolution
+
+Vault content paths are resolved as:
+```
+{env_var_value}/{filename}
+```
+
+For example, with `PSAP_FORGE_FOURNOS_CI_SECRET_PATH=/secrets/ci` and content named `kubeconfig`:
+```
+/secrets/ci/kubeconfig
+```
+
+## Vault Validation
+
+The vault system enforces strict validation to ensure security and documentation quality.
+
+### Validation Rules
+
+1. **Vault description required**: Every vault must have a non-empty description
+2. **Content descriptions required**: Every piece of content must be documented
+3. **File existence**: All defined content files must exist on filesystem
+4. **No extra files**: Vault directories cannot contain undefined files
+5. **Environment variables**: Required env vars must be set and point to existing directories
+
+### Validation Levels
+
+**Strict mode** (default): Fails on any validation error
+**Warning mode**: Logs warnings but continues execution
+
+### Using the Validation System
+
+```python
+from projects.core.library.vault import get_vault_manager
+
+# Initialize vault manager
+vault_manager = get_vault_manager()
+
+# Validate a specific vault
+is_valid = vault_manager.validate_vault('psap-forge-ci', strict=True)
+
+# Validate all project vaults
+is_valid = vault_manager.validate_project_vaults('llm_d', strict=True)
+
+# Validate all defined vaults
+is_valid = vault_manager.validate_all_vaults(strict=True)
+```
+
+### Getting Vault Content Paths
+
+```python
+from projects.core.library.vault import get_vault_content_path
+
+# Get full path to vault content
+kubeconfig_path = get_vault_content_path('psap-forge-ci', 'kubeconfig')
+# Returns: Path('/secrets/ci/kubeconfig')
+
+# Access via vault object
+vault = get_vault_manager().get_vault('psap-forge-ci')
+kubeconfig_path = vault.content['kubeconfig'].file_path
+# Returns: Path('/secrets/ci/kubeconfig')
+```
+
+## Environment Setup
+
+### Required Environment Variables
+
+Each vault requires its corresponding environment variable to be set:
+
+```bash
+export PSAP_FORGE_FOURNOS_CI_SECRET_PATH="/path/to/ci/secrets"
+export PSAP_FORGE_NOTIFICATIONS_SECRET_PATH="/path/to/notification/secrets"
+```
+
+### Directory Structure
+
+Vault directories should contain only the files defined in the vault specification:
+
+```
+/path/to/ci/secrets/
+├── kubeconfig          # Defined in vault spec
+├── server_url          # Defined in vault spec
+└── api_token           # Defined in vault spec
+```
+
+Any extra files will cause validation to fail.
+
+## Creating New Vaults
+
+### 1. Create Vault Definition
+
+Create `/vaults/my-new-vault.yaml`:
+
+```yaml
+env_key: MY_VAULT_SECRET_PATH
+description: |
+  Description of what this vault contains and its purpose
+
+content:
+  credential_file:
+    file: credentials.json
+    description: |
+      Service account credentials for accessing external API
+
+  api_key:
+    description: |
+      API key for service authentication
+```
+
+### 2. Set Environment Variable
+
+```bash
+export MY_VAULT_SECRET_PATH="/path/to/my/secrets"
+```
+
+### 3. Create Secret Files
+
+```bash
+mkdir -p /path/to/my/secrets
+echo "secret-key-value" > /path/to/my/secrets/api_key
+cp service-credentials.json /path/to/my/secrets/credentials.json
+```
+
+### 4. Add to Project Requirements
+
+In `projects/my-project/orchestration/vaults.yaml`:
+
+```yaml
+- name: my-new-vault
+  description: "Custom vault for my project"
+```
+
+### 5. Validate
+
+```python
+from projects.core.library.vault import validate_project_vaults
+
+# Validate the new vault
+validate_project_vaults('my-project', strict=True)
+```
+
+### 6. Expose the env var to the `forge_launcher`
+
+```bash
+forge_launcher config --set-env MY_VAULT_SECRET_PATH "/path/to/my/secrets"
+
+# or, if the environment variable is already set in the terminal
+
+forge_launcher config --pass-env MY_VAULT_SECRET_PATH
+```
+
+## Integration with FORGE Projects
+
+### Initialization
+
+Initialize vault system with specific vaults:
+
+```python
+from projects.core.library import vault
+
+# Initialize with all vaults
+vault.init()
+
+# Initialize with specific vaults only
+vault.init(['psap-forge-ci', 'forge-llm-d'])
+```
+
+### Runtime Usage
+
+```python
+from projects.core.library.vault import get_vault_manager
+
+# Get vault manager
+vault_manager = get_vault_manager()
+
+# Validate project vaults before execution
+if not vault_manager.validate_project_vaults('llm_d'):
+    raise RuntimeError("Vault validation failed")
+
+# Get paths to vault content
+vault = vault_manager.get_vault('forge-llm-d')
+hf_token_path = vault.content['hf_token'].file_path
+pull_secret_path = vault.content['rhoai_pull_secret'].file_path
+```
+
+## Error Handling
+
+Common validation errors and solutions:
+
+**Missing environment variable:**
+```
+Vault 'my-vault' requires environment variable MY_VAULT_SECRET_PATH to be set
+```
+Solution: Export the required environment variable
+
+**Missing description:**
+```
+Vault 'my-vault' content 'api_key' is missing description
+```
+Solution: Add description to vault definition
+
+**Extra files:**
+```
+Vault 'my-vault' contains extra file 'backup.txt' at '/path/vault/backup.txt' not defined in specification
+```
+Solution: Either remove the file or add it to the vault definition
+
+**Missing content file:**
+```
+Vault 'my-vault' missing content 'api_key' at: /path/vault/api_key
+```
+Solution: Create the missing secret file

projects/core/dsl/__init__.py

@@ -6,5 +6,7 @@
 from .runtime import execute_tasks, clear_tasks
 from . import shell
 from . import toolbox
+from . import template
+from . import context
 
-__all__ = ['always', 'clear_tasks', 'execute_tasks', 'retry', 'shell', 'task', 'when']
+__all__ = ['always', 'clear_tasks', 'context', 'execute_tasks', 'retry', 'shell', 'task', 'template', 'when']

projects/core/dsl/context.py

@@ -0,0 +1,79 @@
+"""
+Context management for DSL tasks
+
+Provides readonly args and mutable context for task execution.
+"""
+
+import types
+from typing import Any
+
+
+class ReadOnlyArgs:
+    """
+    A readonly wrapper around the args object that prevents modification
+    """
+
+    def __init__(self, args_dict: dict):
+        # Store the original args in a private attribute
+        object.__setattr__(self, '_args_dict', args_dict)
+
+    def __getattr__(self, name: str) -> Any:
+        args_dict = object.__getattribute__(self, '_args_dict')
+        if name in args_dict:
+            return args_dict[name]
+        raise AttributeError(f"'ReadOnlyArgs' object has no attribute '{name}'")
+
+    def __setattr__(self, name: str, value: Any) -> None:
+        raise AttributeError(f"Cannot modify readonly args. Attempted to set '{name}' = {value}")
+
+    def __delattr__(self, name: str) -> None:
+        raise AttributeError(f"Cannot delete attribute '{name}' from readonly args")
+
+    def __contains__(self, name: str) -> bool:
+        args_dict = object.__getattribute__(self, '_args_dict')
+        return name in args_dict
+
+    def __repr__(self) -> str:
+        args_dict = object.__getattribute__(self, '_args_dict')
+        return f"ReadOnlyArgs({args_dict})"
+
+
+class TaskContext:
+    """
+    Mutable context for values defined within a task
+    """
+
+    def __init__(self):
+        pass
+
+    def __repr__(self) -> str:
+        attrs = {k: v for k, v in self.__dict__.items() if not k.startswith('_')}
+        return f"TaskContext({attrs})"
+
+
+def create_task_parameters(original_args: types.SimpleNamespace, shared_context: types.SimpleNamespace) -> tuple[ReadOnlyArgs, TaskContext]:
+    """
+    Create readonly args and mutable context from the original args object
+
+    Args:
+        original_args: The original args SimpleNamespace from execute_tasks
+        shared_context: The shared context that persists across tasks
+
+    Returns:
+        Tuple of (readonly_args, mutable_context)
+    """
+    # Convert SimpleNamespace to dict for ReadOnlyArgs (only original args, not context)
+    args_dict = vars(original_args)
+
+    readonly_args = ReadOnlyArgs(args_dict)
+
+    # Create a TaskContext that wraps the shared context
+    # This allows the task to modify the shared context
+    context = TaskContext()
+
+    # Copy existing values from shared context to this task's context
+    for attr_name, attr_value in vars(shared_context).items():
+        if not attr_name.startswith('_'):
+            setattr(context, attr_name, attr_value)
+
+    return readonly_args, context