TT-7898 Opentelemetry e2e tests pipeline (#5292)

<!-- Provide a general summary of your changes in the Title above -->

## Description
<!-- Describe your changes in detail -->
Setting up an Opentelemetry e2e testing pipeline on CI that:

1.  Builds a Tyk docker image based on current code
2.  Setup `OpenTelemetry collector`
3.  Setup the `tracetest` environment
4. Execute tests over output traces thru tracetest CLI. The tests are
defined under ci/tests/tracing/scenarios


You can also run the tests manually with `task test:opentelemetry` or
you can execute each step (particularly useful when adding new tests).
If you execute `task -l` you will see all the available commands.

Added a readme for detailed components instruction on
`ci/tests/tracing/README.md`

This PR also includes Test cases for:
- gRPC API + unary (tyk_grpcapi_200.yml)
- HTTP API ok response (tyk_test_200.yml)
- HTTP API Upstream error (tyk_test_500.yml)
- HTTP API Auth middleware error (tyk_testauth_401.yml)
- Tyk protocol API proxying to a open API (non-secured)
(tyk_tykprotocol_200.yml)
- Tyk protocol API proxying to a secured API (401 error)
(tyk_tykprotocol_401.yml)
## Related Issue

<!-- This project only accepts pull requests related to open issues. -->
<!-- If suggesting a new feature or change, please discuss it in an
issue first. -->
<!-- If fixing a bug, there should be an issue describing it with steps
to reproduce. -->
<!-- OSS: Please link to the issue here. Tyk: please create/link the
JIRA ticket. -->
https://tyktech.atlassian.net/browse/TT-7898
## Motivation and Context

<!-- Why is this change required? What problem does it solve? -->
https://tyktech.atlassian.net/browse/TT-7898
## How This Has Been Tested

<!-- Please describe in detail how you tested your changes -->
<!-- Include details of your testing environment, and the tests -->
<!-- you ran to see how your change affects other areas of the code,
etc. -->
<!-- This information is helpful for reviewers and QA. -->
CI :D 
## Screenshots (if appropriate)

## Types of changes

<!-- What types of changes does your code introduce? Put an `x` in all
the boxes that apply: -->

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Refactoring or add test (improvements in base code or adds test
coverage to functionality)

## Checklist

<!-- Go over all the following points, and put an `x` in all the boxes
that apply -->
<!-- If there are no documentation updates required, mark the item as
checked. -->
<!-- Raise up any additional concerns not covered by the checklist. -->

- [ ] I ensured that the documentation is up to date
- [ ] I explained why this PR updates go.mod in detail with reasoning
why it's required
- [ ] I would like a code coverage CI quality gate exception and have
explained why

.github/workflows/opentelemetry-e2e.yml

@@ -0,0 +1,28 @@
+name: Opentelemetry e2e
+
+on:
+  pull_request:
+    branches: [master]
+
+jobs:
+  e2e:
+    name: Opentelemetry e2e
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v2
+      - name: Install Task
+        uses: arduino/setup-task@v1
+        with:
+          version: 3
+      - name: Setup Golang
+        uses: actions/setup-go@v4
+        with:
+          go-version: 1.19  
+      - name: Setup e2e testing enviroment
+        run: task test:opentelemetry:setup
+      - name: Run e2e opentelemetry tests
+        run: task test:opentelemetry:test
+      - name: Stop e2e
+        if: always()
+        run: task test:opentelemetry:teardown

.gitignore

@@ -55,3 +55,4 @@ tyk_linux_*
 /dist/
 
 .terraform**
+.task/

.taskfiles/deps_darwin.yml

@@ -0,0 +1,10 @@
+version: '3'
+
+tasks:
+  tracetest:
+    desc: "install tracetest on darwin"
+    status:
+      - type tracetest
+    cmds:
+      - brew install kubeshop/tracetest/tracetest
+      - tracetest configure -g --endpoint http://localhost:11633

.taskfiles/deps_linux.yml

@@ -0,0 +1,10 @@
+version: '3'
+
+tasks:
+  tracetest:
+    desc: "install tracetest on linux"
+    status:
+      - type tracetest
+    cmds:
+      - curl -L https://raw.githubusercontent.com/kubeshop/tracetest/main/install-cli.sh | bash
+

Dockerfile

@@ -58,7 +58,7 @@ RUN find /tmp -type f -delete
 # Build gateway
 RUN mkdir /opt/tyk-gateway
 WORKDIR /opt/tyk-gateway
-ADD go.mod go.sum /opt/tyk-gateway
+ADD go.mod go.sum /opt/tyk-gateway/
 RUN go mod download
 ADD . /opt/tyk-gateway
 

Taskfile.yml

@@ -1,7 +1,20 @@
 ---
 version: "3"
 
+includes:
+  deps: ./.taskfiles/deps_{{OS}}.yml
+
 tasks:
+  docker:
+    desc: "build Tyk gateway internal/tyk-gateway"
+    cmds:
+      - docker build --platform "linux/amd64" --rm -t internal/tyk-gateway .
+    sources:
+      - go.mod
+      - go.sum
+      - './**/*.go'
+
+
   test:plugin-compiler:
     desc: "Plugin compiler local build/test"
     cmds:
@@ -25,3 +38,42 @@ tasks:
         yq -i ".dockers[0].extra_files |= [\"$replacement\"]" ci/goreleaser/goreleaser-el7.yml
         yq -i ".dockers[4].extra_files |= [\"$replacement\"]" ci/goreleaser/goreleaser.yml
       - echo goreleaser release --clean --snapshot -f ci/goreleaser/goreleaser.yml
+
+  test:opentelemetry:setup:
+    desc: "setup e2e opentelemetry tests"
+    deps: 
+      - deps:tracetest
+      - docker
+    status:
+      - tracetest version
+    cmds:
+      - docker compose -f ci/tests/tracing/docker-compose.yml up -d
+      - tracetest configure -g --endpoint http://localhost:11633
+
+
+  test:opentelemetry:test:
+    desc: "run e2e opentelemetry tests"
+    deps:
+      - test:opentelemetry:setup
+    cmds:
+      - tracetest test run -d ./ci/tests/tracing/scenarios/tyk_test_200.yml -w -o pretty
+      - tracetest test run -d ./ci/tests/tracing/scenarios/tyk_test_500.yml -w -o pretty
+      - tracetest test run -d ./ci/tests/tracing/scenarios/tyk_testauth_401.yml -w -o pretty
+      - tracetest test run -d ./ci/tests/tracing/scenarios/tyk_tykprotocol_200.yml -w -o pretty
+      - tracetest test run -d ./ci/tests/tracing/scenarios/tyk_tykprotocol-auth_401.yml -w -o pretty
+      - tracetest test run -d ./ci/tests/tracing/scenarios/tyk_grpcapi_200.yml -w -o pretty
+
+
+  test:opentelemetry:teardown:
+    desc: "teardown e2e opentelemetry tests enviroment"
+    deps: [test:opentelemetry:setup]
+    cmds:
+      - docker compose -f ci/tests/tracing/docker-compose.yml down --remove-orphans
+
+  test:opentelemetry:
+    desc : "setup, execute and shutdown e2e opentelemetry tests"
+    cmds:
+      - task: test:opentelemetry:setup
+      - task: test:opentelemetry:test
+      - task: test:opentelemetry:teardown
+

ci/tests/tracing/README.md

@@ -0,0 +1,26 @@
+---
+title: Opentelemetry e2e Tests
+---
+
+# Opentelemetry e2e Tests
+
+This docker-compose file defines a multi-service application including the following components:
+
+- `tyk`: This service runs an internal Tyk gateway. So if you are running this from a cold start, make sure to execute `make docker` at the root of Tyk's repo. The environment variables for the tyk service are loaded from a local file, `./configs/tyk.env`. It uses `/apps` folder to mount the preloaded APIs.
+    - `redis`: This service runs another Redis server (version 4.0, based on Alpine Linux). The Redis server is configured to use an append-only file for data persistence.
+    - `httpbin`: This service runs an HTTP Request & Response Service, which is a simple HTTP server for testing and debugging.
+
+
+
+- `otel-collector`: This service runs an instance of the OpenTelemetry Collector (version 0.80.0). It is configured using the file /otel-local-config.yml, which is mounted from the local file ./tracetest/collector.config.yml.
+
+- `tracetest`: This service runs an application from the Docker image kubeshop/tracetest:v0.11.16. It depends on the otel-collector and postgres services. It uses a provisioning file located at `/configs/tracetest/tracetest-provision.yml` and a configuration file located at `/configs/tracetest/tracetest.yml`.
+It needs the following services to work:
+
+    - `postgres`: This service runs a PostgreSQL database (version 14). It uses environment variables to set the PostgreSQL user and password. The health of the PostgreSQL service is checked by running the pg_isready command, which checks the connection to the database server.
+    - `queue`: This service runs a RabbitMQ message broker, with the management plugin enabled. It checks the health of the RabbitMQ server using the rabbitmq-diagnostics check_running command.
+    - `cache`: This service runs a Redis server (version 6) as a cache, with the unless-stopped restart policy. It periodically checks the health of the Redis server using the redis-cli ping command.
+
+
+The `tests` folder contains all the `*.yml` declarations of tests that are going to be executed. Please take into consideration:
+- the target URL for gateway APIs is `http://httpbin:80/` for HTTP APIs