+
+
+
Fig. 4 Workflow of Pushing an Artifact using the General Processor
+ + + +## Out-of-tree processor + +Some contributions have been made about out-of-tree processor: - The new Processor struct `HTTPProcessor` to support artifact processor extender feature for extending custom artifact types. The current design of `Processor` interface is not changed at the same time, thus the new feature will not affect the existing supported types like OCI Image, CNAB and Helm Chart. @@ -168,7 +307,7 @@ To address these problems, we propose a new feature **artifact processor extende - The similar mechanism to Scanner to support Auth in `HTTPProcessor`, which will be used to pull manifests from the Registry. -### HTTPProcessor and Processor Extender +#### HTTPProcessor and Processor Extender The Processor interface is defined in Harbor Core and we do not propose any change for it. @@ -189,7 +328,7 @@ type Processor interface { } ``` -We propose a new implementation of the `Processor` interface, `HTTPProcessor`. The design of the processor is shown in Fig. 4. The processor acts as a proxy to the processor extender. There are two new components in the architecture: +We propose a new implementation of the `Processor` interface, `HTTPProcessor`. The design of the processor is shown in Fig. 5. The processor acts as a proxy to the processor extender. There are two new components in the architecture: 1. Remote Processor API - HTTP RESTful API between Harbor and remote processor. - The API itself is defined and maintained by Harbor. @@ -200,7 +339,7 @@ We propose a new implementation of the `Processor` interface, `HTTPProcessor`. T
-
Fig. 4 Design of HTTPProcessor
+Fig. 5 Design of HTTPProcessor
For a remote processor, the functions defined in ```Processor``` interface can be abstract to HTTP service API. By using these APIs, Harbor Core can communicate with remote HTTP processor extender. @@ -252,11 +391,11 @@ func (h *HTTPProcessor) AbstractAddition(ctx context.Context, artifact *artifact } ``` -The workflow of pushing a custom artifact to Harbor with the help of `HTTPProcessor` is shown in Fig. 5. The Harbor works as a proxy to the registry when the user uploads the content layers and config layer. Harbor ensures that the repository exists. Then Harbor puts the manifest to the registry. After that, Harbor will check if the artifact by digest exists. In this step, Harbor will use `Processor.AbstractMetatda` in `Abstractor` to abstract the metadata ant keep in the artifact.Artifact model. +The workflow of pushing a custom artifact to Harbor with the help of `HTTPProcessor` is shown in Fig. 6. The Harbor works as a proxy to the registry when the user uploads the content layers and config layer. Harbor ensures that the repository exists. Then Harbor puts the manifest to the registry. After that, Harbor will check if the artifact by digest exists. In this step, Harbor will use `Processor.AbstractMetatda` in `Abstractor` to abstract the metadata ant keep in the artifact.Artifact model.
-
Fig. 5 Workflow of Pushing an Artifact using the HTTPProcessor
+Fig. 6 Workflow of Pushing an Artifact using the HTTPProcessor
When `HTTPProcessor.AbstractMetadata(ctx context.Context, manifest []byte, artifact *artifact.Artifact) error)` is invoked, it will send a HTTP POST request to the processor extender: @@ -277,11 +416,11 @@ POST {remote-processor-endpoint}/abstractmetadata The `manifest` and `artifact` will be sent to the extender to abstract the metadata. If the registry needs Auth, `registry` will be sent, just like Scanner. -When users deal with these built-in artifact types like OCI Image, CNAB or Helm Chart, the workflow **is not affected**, like Fig. 6. For example, the user uploads a Helm Chart to Harbor, we will use Helm Chart processor to abstract the metadata. Thus the design of `HTTPProcessor` is non-invasive to the current design. +When users deal with these built-in artifact types like OCI Image, CNAB or Helm Chart, the workflow **is not affected**, like Fig. 7. For example, the user uploads a Helm Chart to Harbor, we will use Helm Chart processor to abstract the metadata. Thus the design of `HTTPProcessor` is non-invasive to the current design.
-
Fig. 6 Workflow of Pushing an Artifact using the Build-in Processor
+Fig. 7 Workflow of Pushing an Artifact using the Build-in Processor
#### Processor Extender @@ -415,7 +554,7 @@ It is reasonable to use the same way for processor extender using bearer tokens Refer to scan job using credentials generated by robot account mechanism, we can use the same way to use the robot account mechanism to generate credentials that work with these common OCI/Docker tooling libraries to provide credentialed access to the image data. The lifecycle of the robot account credentials can be bound to the HTTP request. For every HTTP request call remote processor API, a robot account expired at certain time will be created.Additionally, a modification is needed to ensure that the generated credentials have access to bypass the configured policy checks on the image that normal users are subject to if those checks are configured. -## Development Process +### Development Process There are totally three things we need to do to complete the proposal: @@ -425,16 +564,16 @@ There are totally three things we need to do to complete the proposal: Thus we propose to have three iterations. Each of them is self-contained and supposed to be merged into Harbor Core. -### First Iteration: HTTPProcessor and Extender without Auth +#### First Iteration: HTTPProcessor and Extender without Auth At this stage, user defined processor will not register to harbor. So if users want to use processor extender, they still need to hard-code some logic to harbor code to register the type with the corresponding processor manually. Also, ```HTTPProcessor``` will make HTTP request to the extender without providing authentication and Harbor external endpoint. So users need to do some work to generate authentication using other user account. Harbor external endpoint should be configured any way. And policy check interceptor can not be bypassed. -### Second Iteration: Registration +#### Second Iteration: Registration At the second stage, registration logic will be added. Users don't need to modify harbor code any more. A remote processor configuration file is required to register specific processor to harbor. When harbor core starts, it will read the configuration file and register the processor to harbor. -### Third Iteration: Auth in the Processor Extender +#### Third Iteration: Auth in the Processor Extender At the final stage, using robot account mechanism to generate credentials will be finished. Harbor external endpoint and authentication will be passed directly in HTTP POST request body. Users don't need to worry about the authentication problem. But still need to find a way to use authentication properly. diff --git a/proposals/images/artifact-processor-extender/in-tree-workflow.png b/proposals/images/artifact-processor-extender/in-tree-workflow.png new file mode 100644 index 00000000..34a6c05e Binary files /dev/null and b/proposals/images/artifact-processor-extender/in-tree-workflow.png differ diff --git a/workgroups/.DS_Store b/workgroups/.DS_Store new file mode 100644 index 00000000..2d544598 Binary files /dev/null and b/workgroups/.DS_Store differ