test(e2e): add E2E test infra (#561)

This PR:
1. Adds distribution config and cred file for basic auth
2. Runs ORAS distribution, installs and runs Ginkgo in build CI
4. Added a sample test to make the added CI step pass
5. Added mount files to initialize test data for command suite

Resolves #552, resolves #553

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

.github/licenserc.yml

@@ -34,10 +34,16 @@ header:
     - 'KEYS'
     - 'go.mod'
     - 'go.sum'
+    - 'go.work'
     - '**/testdata/**'
 
   comment: on-failure
 
 dependency:
   files:
     - go.mod
+  licenses:
+    # known issue: https://github.com/apache/skywalking-eyes/pull/107#issuecomment-1129761574
+    - name: github.com/chzyer/logex
+      version: v1.1.10
+      license: MIT

.github/workflows/build.yml

@@ -39,6 +39,25 @@ jobs:
         run: make build-linux-amd64
       - name: Run Unit Tests
         run: make test
+      - name: Run E2E Tests
+        run: |
+          cd $GITHUB_WORKSPACE/test/e2e
+          mnt_root="$GITHUB_WORKSPACE/test/e2e/testdata/distribution/mount"
+          go install github.com/onsi/ginkgo/v2/ginkgo
+          rm -rf $mnt_root/docker
+          for layer in $(ls $mnt_root/*.tar.gz); do
+            tar -xvzf $layer -C $mnt_root
+          done
+          trap 'docker kill oras-e2e || true' ERR
+          docker run -d -p 5000:5000 --rm --name oras-e2e \
+            --env STORAGE_DELETE_ENABLED=true \
+            --mount type=bind,source=$mnt_root/docker,target=/opt/data/registry-root-dir/docker \
+            ghcr.io/oras-project/registry:v1.0.0-rc.2
+          ginkgo -r -p suite
+          docker kill oras-e2e || true
+        env:
+          ORAS_PATH: bin/linux/amd64/oras
+          ORAS_REGISTRY_HOST: localhost:5000
       - name: Check Version
         run: bin/linux/amd64/oras version
       - name: Upload coverage to codecov.io

.gitignore

@@ -38,3 +38,6 @@ dist/
 *.tar.gz
 vendor/
 _dist/
+
+# Distribution storage files for local E2E testing
+test/e2e/testdata/distribution/mount/docker/ 

test/e2e/README.md

@@ -0,0 +1,119 @@
+# ORAS End-to-End Testing Dev Guide
+
+## Setting up
+Minimal setup: Run the script in **step 3**
+
+### 1. Clone Source Code of ORAS CLI
+```shell
+git clone https://github.com/oras-project/oras.git
+```
+
+### 2. _[Optional]_ Install Ginkgo
+This will enable you use `ginkgo` directly in CLI.
+```shell
+go install github.com/onsi/ginkgo/v2/ginkgo@latest
+```
+If you skip step 2, you can only run tests via `go test`. 
+
+### 3. Run Distribution
+The backend of E2E test is an [oras-distribution](https://github.com/oras-project/distribution).
+```shell
+PORT=5000
+docker run -dp $PORT:5000 --rm --name oras-e2e \
+    --env STORAGE_DELETE_ENABLED=true \
+    ghcr.io/oras-project/registry:v1.0.0-rc.2
+```
+
+### 4. _[Optional]_ Customize Port for Distribution
+```shell
+export ORAS_REGISTRY_HOST="localhost:$PORT"
+# for PowerShell, use $env:ORAS_REGISTRY_HOST = "localhost:$PORT"
+```
+If you skipped step 4, E2E test will look for distribution ran in `localhost:5000`
+
+### 5. _[Optional]_ Setup ORAS Binary for Testing
+```bash
+# Set REPO_ROOT as root folder of oras CLI code
+cd $REPO_ROOT
+make build
+```
+### 6. _[Optional]_ Setup Pre-Built Binary
+You need to setup below environmental variables to debug a pre-built ORAS binary:
+```bash
+export ORAS_PATH="bin/linux/amd64/oras" # change target platform if needed
+export GITHUB_WORKSPACE=$REPO_ROOT
+```
+If you skipped step 5 or 6, Gomega will build a temp binary, which will include all the CLI code changes in the working directory.
+
+### 7. _[Optional]_ Mount Test Data
+If you want to run command suite, you need to decompress the registry storage files and mount to the distribution. `$REPO_ROOT` points to the root folder of cloned oras CLI code.
+```shell
+mnt_root=${REPO_ROOT}/test/e2e/testdata/distribution/mount
+for layer in $(ls ${mnt_root}/*.tar.gz); do
+    tar -xvzf $layer -C ${mnt_root}
+done
+
+PORT=5000
+docker run -dp ${PORT}:5000 --rm --name oras-e2e \
+    --env STORAGE_DELETE_ENABLED=true \
+    --mount type=bind,source=${mnt_root}/docker,target=/opt/data/registry-root-dir/docker \
+    ghcr.io/oras-project/registry:v1.0.0-rc.2
+```
+Skipping step 7 you will not be able to run specs in Command suite.
+
+## Development
+### 1. Constant Build & Watch
+This is a good choice if you want to debug certain re-runnable specs
+```bash
+cd $REPO_ROOT/test/e2e
+ginkgo watch -r
+```
+
+### 2. Debugging
+Since E2E test suites are added to a sub-module, you need to run `go test` from `$REPO_ROOT/test/e2e/`. If you need to debug a certain spec, use [focused spec](https://onsi.github.io/ginkgo/#focused-specs) but don't check it in.
+
+### 3. Trouble-shooting CLI
+Executed command should be shown in the ginkgo logs after `[It]`,
+
+### 4. Adding New Tests
+Two suites will be maintained for E2E testing:
+- command: contains test specs for single oras command execution
+- scenario: contains featured scenarios with several oras commands execution
+
+Inside a suite, please follow below model when building the hierarchical collections of specs:
+```
+Describe: <Role>
+  Context: Scenario or command specific description
+    When: <Action>
+      It: <Result> (per-command execution)
+       Expect: <Result> (detailed checks for execution results)
+```
+
+### 5. Adding New Test Data
+
+#### 5.1 Command Suite
+Command suite uses pre-baked registry data for testing. The repository name should be `command/$repo_suffix`. To add a new layer, compress the `docker` folder from the root directory of your distribution storage and copy it to `$REPO_ROOT/test/e2e/testdata/distribution/mount` folder.
+```shell
+tar -cvzf ${repo_suffix}.tar.gz --owner=0 --group=0 docker/
+```
+Currently we have below OCI images:
+```mermaid
+graph TD;
+    subgraph images.tar.gz
+    A0>tag: multi]-..->A1[oci index]
+    A1--linux/amd64-->A2[oci image]
+    A1--linux/arm64-->A3[oci image]
+    A1--linux/arm/v7-->A4[oci image]
+    A2-->A5[config1]
+    A3-->A6[config2]
+    A4-->A7[config3]
+    A2-- hello.tar -->A8[blob]
+    A3-- hello.tar -->A8[blob]
+    A4-- hello.tar -->A8[blob]
+
+    B0>tag: foobar]-..->B1[oci image]
+    B1-- foo1 -->B2[blob1]
+    B1-- foo2 -->B2[blob1]
+    B1-- bar -->B3[blob2]
+    end
+```

test/e2e/go.mod

@@ -0,0 +1,20 @@
+module oras.land/oras/test/e2e
+
+go 1.19
+
+require (
+	github.com/onsi/ginkgo/v2 v2.1.6
+	github.com/onsi/gomega v1.20.2
+	oras.land/oras-go/v2 v2.0.0-rc.3.0.20220922092058-3f9653f7bf69
+)
+
+require (
+	github.com/google/go-cmp v0.5.8 // indirect
+	github.com/opencontainers/go-digest v1.0.0 // indirect
+	github.com/opencontainers/image-spec v1.0.2 // indirect
+	github.com/oras-project/artifacts-spec v1.0.0-rc.2 // indirect
+	golang.org/x/net v0.0.0-20220722155237-a158d28d115b // indirect
+	golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f // indirect
+	golang.org/x/text v0.3.7 // indirect
+	gopkg.in/yaml.v3 v3.0.1 // indirect
+)

test/e2e/go.sum

@@ -0,0 +1,27 @@
+github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
+github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg=
+github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
+github.com/onsi/ginkgo/v2 v2.1.6 h1:Fx2POJZfKRQcM1pH49qSZiYeu319wji004qX+GDovrU=
+github.com/onsi/ginkgo/v2 v2.1.6/go.mod h1:MEH45j8TBi6u9BMogfbp0stKC5cdGjumZj5Y7AG4VIk=
+github.com/onsi/gomega v1.20.2 h1:8uQq0zMgLEfa0vRrrBgaJF2gyW9Da9BmfGV+OyUzfkY=
+github.com/onsi/gomega v1.20.2/go.mod h1:iYAIXgPSaDHak0LCMA+AWBpIKBr8WZicMxnE8luStNc=
+github.com/opencontainers/distribution-spec/specs-go v0.0.0-20220620172159-4ab4752c3b86 h1:Oumw+lPnO8qNLTY2mrqPJZMoGExLi/0h/DdikoLTXVU=
+github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
+github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
+github.com/opencontainers/image-spec v1.0.2 h1:9yCKha/T5XdGtO0q9Q9a6T5NUCsTn/DrBg0D7ufOcFM=
+github.com/opencontainers/image-spec v1.0.2/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0=
+github.com/oras-project/artifacts-spec v1.0.0-rc.2 h1:9SMCNSxkJEHqWGDiMCuy6TXHgvjgwXGdXZZGXLKQvVE=
+github.com/oras-project/artifacts-spec v1.0.0-rc.2/go.mod h1:Xch2aLzSwtkhbFFN6LUzTfLtukYvMMdXJ4oZ8O7BOdc=
+golang.org/x/net v0.0.0-20220722155237-a158d28d115b h1:PxfKdU9lEEDYjdIzOtC4qFWgkU2rGHdKlKowJSMN9h0=
+golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
+golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f h1:v4INt8xihDGvnrfjMDVXGxw9wrfxYyCjk0KbXjhR55s=
+golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
+golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
+google.golang.org/protobuf v1.28.0 h1:w43yiav+6bVFTBQFZX0r7ipe9JQ1QsbMgHwbBziscLw=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+oras.land/oras-go/v2 v2.0.0-rc.3.0.20220922092058-3f9653f7bf69 h1:HpwQf1GbCkpVTdmHSsBiJpfZMWxqPjPjcKbtX6VXtGI=
+oras.land/oras-go/v2 v2.0.0-rc.3.0.20220922092058-3f9653f7bf69/go.mod h1:PrY+cCglzK/DrQoJUtxbYVbL94ZHecVS3eJR01RglpE=

test/e2e/go.work

@@ -0,0 +1,6 @@
+go 1.19
+
+use (
+	.
+	../..
+)

test/e2e/internal/utils/init.go

@@ -0,0 +1,68 @@
+/*
+Copyright The ORAS Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package utils
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+
+	. "github.com/onsi/ginkgo/v2"
+	"github.com/onsi/gomega"
+	"github.com/onsi/gomega/gexec"
+	"oras.land/oras-go/v2/registry"
+)
+
+// ORASPath points to the to-be-tested oras binary.
+var ORASPath string
+
+// Host points to the registry service where E2E tests will be run against.
+var Host string
+
+func init() {
+	Host = os.Getenv("ORAS_REGISTRY_HOST")
+	if Host == "" {
+		Host = "localhost:5000"
+		fmt.Fprintln(os.Stderr, "cannot find host name in ORAS_REGISTRY_HOST, using", Host, "instead")
+	}
+	ref := registry.Reference{
+		Registry: Host,
+	}
+	if err := ref.ValidateRegistry(); err != nil {
+		panic(err)
+	}
+	BeforeSuite(func() {
+		ORASPath = os.Getenv("ORAS_PATH")
+		if filepath.IsAbs(ORASPath) {
+			fmt.Printf("Testing based on pre-built binary locates in %q\n", ORASPath)
+			return
+		}
+
+		var err error
+		if workspacePath := os.Getenv("GITHUB_WORKSPACE"); ORASPath != "" && workspacePath != "" {
+			// add workspacePath as prefix, both path env should not be empty
+			ORASPath = filepath.Join(workspacePath, ORASPath)
+			ORASPath, err = filepath.Abs(ORASPath)
+			gomega.Expect(err).NotTo(gomega.HaveOccurred())
+			fmt.Printf("Testing based on pre-built binary locates in %q\n", ORASPath)
+			return
+		}
+
+		// fallback to native build to facilitate local debugging
+		ORASPath, err = gexec.Build("oras.land/oras/cmd/oras")
+		gomega.Expect(err).NotTo(gomega.HaveOccurred())
+		DeferCleanup(gexec.CleanupBuildArtifacts)
+		fmt.Printf("Testing based on temp binary locates in %q\n", ORASPath)
+	})
+}

test/e2e/suite/command/command_suite_test.go

@@ -0,0 +1,28 @@
+/*
+Copyright The ORAS Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package command
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestCommand(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "Command Suite")
+}

test/e2e/suite/command/version.go

@@ -0,0 +1,37 @@
+/*
+Copyright The ORAS Authors.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package command
+
+import (
+	"os/exec"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"github.com/onsi/gomega/gexec"
+	"oras.land/oras/test/e2e/internal/utils"
+)
+
+var _ = Describe("ORAS user", func() {
+	Context("checks oras version", func() {
+		When("running version command", func() {
+			It("should success", func() {
+				session, err := gexec.Start(exec.Command(utils.ORASPath, "version"), nil, nil)
+				Expect(err).ShouldNot(HaveOccurred())
+				Eventually(session, "10s").Should(gexec.Exit(0))
+			})
+		})
+	})
+})