Job template

[alexec]

Summary

Workflows is used by some as a CI solution. Yet it is really needs a new template type to be comparable to GitHub Actions.

The template type could be called “job”. A job runs a series of steps. Each step runs a command (including a shell script) and the job fails if the command fails. Steps execute in sequence. The job fails if any step fails. Steps execute conditionally, usually only if the job is passing.

How’s this different to a steps template? Each step of a job runs in the same container. This makes storing and restoring build and caches faster and simpler.

It also allows users to visualise steps within a container.

Code changes will be similar to adding the container-set template.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: ci-v2
  annotations:
    workflows.argoproj.io/description: |
      This workflows builds and tests Argo Workflows. 
      
      It demostrates:
      
      * Cache restore and store.
      * Publishing test reports.
spec:
  arguments:
    parameters:
      - name: branch
        value: master

  entrypoint: main

  templates:
    - name: main
      inputs:
        artifacts:
          - name: GOMODCACHE
            path: /go/pkg/mod
            optional: true
            s3:
              key: "github.com/golang/examples/{{workflow.parameters.branch}}/GOMODCACHE"
          - name: GOCACHE
            path: /root/.cache/go-build
            optional: true
            s3:
              key: "github.com/golang/examples/{{workflow.parameters.branch}}/GOCACHE"
      job:
        image: golang:1.18
        workingDir: /go/src/github.com/golang/example
        steps:
          - run: git clone -v -b "{{workflow.parameters.branch}}" --single-branch --depth 1 https://github.com/golang/example.git .
          - run: go mod download -x
          - run: go build ./...
          - run: |
              go install github.com/jstemmer/go-junit-report@latest
              go install github.com/alexec/junit2html@latest
              
              trap 'cat test.out | go-junit-report | junit2html > test-report.html' EXIT
              
              go test -v ./... 2>&1 > test.out
      outputs:
        artifacts:
          - name: test-report
            path: /go/src/github.com/golang/example/test-report.html
            archive:
              none: { }
            s3:
              key: "{{workflow.parameters.branch}}/test-report.html"
          - name: GOMODCACHE
            path: /go/pkg/mod
            optional: true
            s3:
              key: "github.com/golang/examples/{{workflow.parameters.branch}}/GOMODCACHE"
          - name: GOCACHE
            path: /root/.cache/go-build
            optional: true
            s3:
              key: "github.com/golang/examples/{{workflow.parameters.branch}}/GOCACHE"

Use Cases

When would you use this?

---

Message from the maintainers:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

[or-shachar]

@alexec I like the direction!

I think we need to also enhance the cache restore/save to be according to files:

      job:
        image: golang:1.18
        workingDir: /go/src/github.com/golang/example
        steps:
          - run: git clone -v -b "{{workflow.parameters.branch}}" --single-branch --depth 1 https://github.com/golang/example.git .
          - get_artifact:
              name: GOMODCACHE
              path: /go/pkg/mod
              optional: true
              s3:
                key: "github.com/golang/examples/{{ hash 'go.mod' }}/GOMODCACHE"
          - run: go mod download -x
          - save_artifact:
              name: GOMODCACHE
              path: /go/pkg/mod
              optional: true
              s3:
                key: "github.com/golang/examples/{{ hash 'go.mod' }}/GOMODCACHE"
          - run: go build ./...
          - run: |
              go install github.com/jstemmer/go-junit-report@latest
              go install github.com/alexec/junit2html@latest
              
              trap 'cat test.out | go-junit-report | junit2html > test-report.html' EXIT
              
              go test -v ./... 2>&1 > test.out

[alexec]

I've spend a few hours today advancing my PoC. It is ready for alpha testing.

• Read the proposal so you get an understanding of it.
• Update your the image tags in your deployment (workflow-controller, argo-server) to use :dev-jobs rather than :latest. E.g. quay.io/argoproj/workflow-controller:dev-jobs.
• Update your workflow-controller deployment to add --executor-image=quay.io/argoproj/argoexec:dev-jobs
• Create a new job template kubectl apply - f https://github.com/argoproj/argo-workflows/blob/dev-jobs/examples/job-workflowtemplate.yaml.
• Run the the template from the user interface (I don't have a CLI for you yet).

Testing out the feature will demonstrate that users are interested in it.

[alexec]

I forgot, here is a screenshot in action:

[tooptoop4]

i tink can close as can achieve this already