Introduce extensibility to Chainloop CLI

[gr0]

Introduce extensibility to Chainloop CLI

Summary

This issue proposes the implementation of a plugin engine that allows the main CLI application to dynamically load and execute commands from external plugins that. The plugin system enables extensibility without modifying the core application.

Goals

• Enable dynamic command loading without core application changes
• Provide a clean, stable API for plugin development
• Support multiple plugins without conflicts
• Maintain security and isolation between plugins and the main application
• Support for major OS - Linux/Windows and MacOS

Detailed Design

Overview

The plugin engine is built on HashiCorp's go-plugin framework, providing a RPC-based communication layer between the Chainloop CLI application and plugins.

The logical view of the whole architecture looks as follows:

------------------------------------------------------------------
|                Chainloop CLI Application                       |
|----------------------------------------------------------------|
|  -------------------  -------------------  ------------------- |
|  |   Built-in      |  |   Plugin        |  |   Command       | |
|  |   Commands      |  |   Manager       |  |   Router        | |
|  -------------------  -------------------  ------------------- |
|----------------------------------------------------------------|
|                    Plugin Engine                           |
|  -------------------  -------------------  ------------------- |
|  |   Plugin        |  |   RPC           |  |   Command       | |
|  |   Loader        |  |   Communication |  |   Registry      | |
|  -------------------  -------------------  ------------------- |
|----------------------------------------------------------------|
|                      RPC Layer                                 |
------------------------------------------------------------------
                               |
                               | Unix Sockets / Named Pipes
                               |
------------------------------------------------------------------
|                    Plugin Processes                            |
|  -------------------  -------------------  ------------------- |
|  |   Plugin A      |  |   Plugin B      |  |   Plugin C      | |
|  -------------------  -------------------  ------------------- |
------------------------------------------------------------------

Plugin Discovery and Storage

Plugin Directory Structure

~/.config/chainloop/plugins/
  |---- plugin-a* 
  |---- plugin-c*             

Discovery Process

The following process should be a part of the plugin discovery:

1. Scan Directory: The plugin manager scans ~/.config/chainloop/plugins/ for executable files.
2. Validate Executables: Checks file permissions and executable bit (Unix) or .exe extension (Windows)
3. Handshake: Performs plugin handshake to verify compatibility
4. Metadata Extraction: Retrieves plugin metadata including commands, version, and description

Plugin Communication Protocol

RPC Interface Definition

type Plugin interface {
    // Exec executes a command within the plugin
    Exec(ctx context.Context, command string, arguments map[string]interface{}) (ExecResult, error)
    
    // GetMetadata returns plugin metadata including commands it provides
    GetMetadata(ctx context.Context) (PluginMetadata, error)
}

type ExecResult interface {
    GetOutput() string       // Command output for display
    GetError() string        // Error message if execution failed
    GetExitCode() int        // Process exit code
    GetData() map[string]interface{} // Structured data for programmatic use
}

type PluginMetadata struct {
    Name        string        `json:"name"`
    Version     string        `json:"version"`
    Description string        `json:"description"`
    Commands    []CommandInfo `json:"commands"`
}

type CommandInfo struct {
    Name        string     `json:"name"`
    Description string     `json:"description"`
    Usage       string     `json:"usage"`
    Flags       []FlagInfo `json:"flags"`
}

Communication Flow

The communication from between the Chainloop CLI will look as follows:

1. Plugin Startup: Chainloop CLI application spawns plugin process
2. Handshake: Establishes RPC connection with magic cookie validation
3. Metadata Query: Retrieves available commands and their specifications
4. Command Registration: Registers plugin commands with the main CLI router
5. Command Execution: Routes user commands to appropriate plugins via RPC
6. Response Handling: Processes plugin responses and displays results
7. Cleanup: Gracefully shuts down plugin processes on application exit

Conflict Resolution Strategy

It may happen that two distinct plugins expose the same command extension. In such case, in the initial implementation the main Chainloop CLI will fail. Later on we can expand the functionality with command overwrite or configurable behavior.

Environment and Configuration Injection

The main Chainloop CLI app will automatically forward relevant environment variables to plugins. For example using an arguments map like this:

func injectEnvironment(arguments map[string]interface{}) {
    if val := os.Getenv("SOME_ENV_VAR"); val != "" {
        arguments["plugin_expected_var"] = val
    }
}

On the plugin side the it would be used more or less as follows:

func (p *ExamplePlugin) Exec(ctx context.Context, command string, arguments map[string]interface{}) (plugins.ExecResult, error) {
	switch command {
	case "some-command":
		return p.execSomeCommand(ctx, arguments)
	default:
		return &Result{
			Error:    fmt.Sprintf("Unknown command: %s", command),
			ExitCode: 1,
		}, nil
	}
}

func (p *ExamplePlugin) Exec(ctx context.Context, arguments map[string]interface{}) (plugins.ExecResult, error) {
  return &Result{
    Output: "some output string",
    Data: map[string]interface{}{
      "info_one": "some information",
      "info_two": "second information"
    },
  }, 
  nil
}

// type implementing the plugins.ExecResult interface
type Result struct {
	Output   string
	Error    string
	ExitCode int
	Data     map[string]interface{}
}

This should allow to pass all the necessary information to the plugin itself and back to the main Chainloop CLI and getting the response back without the need of changing the plugin interfaces with each new requirements from the Chainloop CLI side or the plugin side.

Core Management Interface

Chainloop main CLI should provide a command that will initially allow:

• listing available plugins
• retrieving given plugin details

In the later stages we should allow:

• downloading a plugin form a configurable destination
• installing a plugin from a given file

Error Handling and Recovery

Plugin Failure Scenarios

Plugins can fail, be incompatible or not work at all, the possible scenarios are:

1. Plugin Crash: Plugin process terminates unexpectedly

   • Response: Ignore plugin and continue with remaining functionality

2. Communication Timeout: Plugin doesn't respond within timeout period

   • Response: Kill plugin process, return timeout error to user

3. Invalid Response: Plugin returns malformed data

   • Response: Log parsing error, return error message

Security Considerations

Process Isolation

• Separate Processes: Each plugin runs in its own process space
• Limited Privileges: Plugins inherit user privileges, not elevated permissions
• Resource Limits: Configurable memory and CPU limits per plugin

Communication Security

• Local-Only: RPC communication uses local Unix sockets/named pipes
• Magic Cookie: Handshake includes magic cookie validation
• Process Ownership: Plugins must be owned by the same user

Implementation Plan

Required Changes

1. Plugin Engine (pkg/plugins/)

The following extension will have to be provided:

New Files:

pkg/plugins/
 |---- interface.go          # Main interfaces
 |---- manager.go           # Plugin manager  
 |---- shared.go              # Handshake and RPC types registration
 |---- client.go                 # RPC client/server implementation

In addition to the above the app/cli/cmd package will have to be extended with the command support.

[gr0]

Let me know what you think.

[migmartri]

Let me know what you think.

oh great writeup, I'll take a look at it asap :)

[migmartri]

Let me know what you think.

very nice writeup, I think it makes sense, there are two main things I am not sure about

Env variables forwarding

I do not think I understand how that works, why it should be forwarded in the first place, and moreover why as an argument.

Persistent flags

How do persistent flags like --api-token, --output, ... and viper configuration would be forwarded?

[gr0]

I was thinking about passing all the parameters as named arguments to the plugin - do you think that would be enough, or maybe too much and the plugin should inform the main cli which flags are required and which ones should be passed through. What do you think?