[Proposal] Use Go Modules for Policy Distribution Instead of Policy Hub Hosting #747
Closed
renuka-fernando
started this conversation in
Ideas
Replies: 3 comments
-
Implementation PlanGateway-Builder ImplementationGateway-Builder TasksPhase 1: Types & Parsing
Phase 2: Go Module Integration
Phase 3: Code Generation Updates
Phase 4: Build Flow Integration
Phase 5: Error Handling
CLI ImplementationCLI TasksPhase 1: Remove PolicyHub Distribution Integration
Phase 2: Simplify Build Flow
Phase 3: Update Manifest Handling
Phase 4: Cleanup Unused Code
Phase 5: Update Tests
Definition of Done
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Issue to track the final implementation: #763 |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
This is completed. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Summary
This proposal replaces Policy Hub's policy hosting and distribution function with standard Go module distribution. Instead of downloading policy zip files from Policy Hub, policies will be fetched using
go getfrom public Git repositories (e.g.,github.com/wso2/gateway-controllers). This eliminates the need to host policy artifacts on Policy Hub while leveraging Go's built-in dependency management, versioning, and integrity verification.Note: Policy Hub continues to exist as a web application for policy discovery and display. This proposal only removes its role as a policy artifact hosting service.
Motivation
Problem Statement
The current policy distribution system requires:
~/.wso2ap/cache/policies/with custom index filesThis adds complexity that can be eliminated by using Go's native module system for policy distribution.
Who Benefits
Why Now
Detailed Design
Overview
The revamp moves policy resolution and download from the CLI to the gateway-builder. The CLI becomes a thin wrapper that runs Docker commands. The gateway-builder uses
go getto fetch policies as Go modules, readspolicy-definition.yamlfrom the Go module cache, generates the necessary code, and compiles the policy engine. Local policies are supported via Go module replace directives.Architecture Comparison
Current Architecture:
flowchart TB M1[policy-manifest.yaml] --> CLI1[CLI] CLI1 -->|POST /policies/resolve| PH[Policy Hub API] PH -->|resolved versions + URLs| CLI1 CLI1 -->|download zips| DL[Download & Cache] DL --> EX[Extract to Workspace] EX --> GB1[Gateway Builder] GB1 -->|read from filesystem| PD1[policy-definition.yaml] GB1 -->|manual update| GM1[go.mod with replace] GB1 --> REG1[Generate registry.go] GB1 --> COMP1[Compile]Proposed Architecture:
flowchart TB M2[policy-manifest.yaml] --> CLI2[CLI] CLI2 -->|docker run| GB2[Gateway Builder] GB2 -->|go get| GOC[Go Module Cache] GB2 -->|go list -m| VER[Get Resolved Version] GB2 -->|read from cache| PD2[policy-definition.yaml] GB2 --> REG2[Generate registry.go] GB2 --> LOCK[Generate lock file] GB2 --> COMP2[Compile]Build Process Flow
sequenceDiagram participant User participant CLI participant Docker participant Builder as Gateway Builder participant Go as Go Toolchain participant Cache as Module Cache User->>CLI: ap gateway build --image-tag v1.0.0 CLI->>CLI: Validate manifest exists CLI->>Docker: docker run gateway-builder Docker->>Builder: Start container Builder->>Builder: Parse policy-manifest.yaml loop For each policy alt Has filePath (local) Builder->>Builder: Add replace directive to go.mod end Builder->>Go: go get <gomodule>@<version> Go->>Cache: Download module Cache-->>Go: Module cached end Builder->>Go: go mod tidy loop For each policy Builder->>Go: go list -m -json <module> Go-->>Builder: {Version, Dir, ...} Builder->>Cache: Read policy-definition.yaml Cache-->>Builder: Policy metadata end Builder->>Builder: Generate policy-manifest-lock.yaml Builder->>Builder: Generate registry.go Builder->>Builder: Generate build_info.go Builder->>Go: go build Go-->>Builder: policy-engine binary Builder->>Builder: Generate Dockerfiles Builder-->>Docker: Exit with artifacts Docker-->>CLI: Build complete CLI->>Docker: docker build (policy-engine) CLI->>Docker: docker build (gateway-controller) CLI->>Docker: docker build (router) opt --push flag CLI->>Docker: docker push (all images) end CLI-->>User: Build complete, images readyPolicy Engine Docker Image Contents
The final policy-engine Docker image includes the following artifacts for traceability and reproducibility:
policy-engine/app/policy-enginego.mod/app/go.modgo.sum/app/go.sumpolicy-manifest-lock.yaml/app/policy-manifest-lock.yamlThis allows:
Changes Required
internal/discovery/manifest.gogomodulefieldinternal/discovery/gomodule.gogo get,go listcommandsinternal/discovery/cache.gointernal/policyengine/gomod.gointernal/policyengine/registry.gointernal/lockfile/generator.gopkg/types/manifest.gocmd/gateway/build.gocmd/gateway/image/build.gointernal/policyhub/internal/policy/hub.goutils/policy_cache.goAPI Changes
N/A - No REST API changes. This affects the build tooling only.
Configuration Changes
policy-manifest.yaml format change:
policy-manifest-lock.yaml format (generated):
Examples
Before (current format):
After (proposed format):
Local policy example:
Version specifier examples:
Drawbacks
policy-manifest.yamlfiles must be updated to new formatGOPRIVATEenvironment variable, or alternatively clone the repo locally and usefilePathCompatibility
name/versionformat no longer supportedMigration Steps
policy-manifest.yamlto new format:gomodulefield with full module path and versionnamefield for identificationfilePathfor any local policiespolicy-manifest-lock.yaml(will be regenerated)rm -rf ~/.wso2ap/cache/policies/ap gateway build- new lock file will be generatedExample migration:
References
Beta Was this translation helpful? Give feedback.
All reactions