From 2fe0043014c169b4d74c39f1c37f6c0edbbb475f Mon Sep 17 00:00:00 2001 From: George Claireaux Date: Thu, 29 Jul 2021 22:23:46 +0100 Subject: [PATCH] Temporary solution for hitting secrets limit (#5085) * add more-secrets environment to publish and test workflows * add link to more-secrets environment in relevant docs location --- .github/workflows/publish-command.yml | 1 + .github/workflows/test-command.yml | 1 + docs/connector-development/README.md | 2 +- 3 files changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/publish-command.yml b/.github/workflows/publish-command.yml index a10f14668305..86def523de0a 100644 --- a/.github/workflows/publish-command.yml +++ b/.github/workflows/publish-command.yml @@ -42,6 +42,7 @@ jobs: publish-image: needs: start-publish-image-runner runs-on: ${{ needs.start-publish-image-runner.outputs.label }} + environment: more-secrets steps: - name: Search for valid connector name format id: regex diff --git a/.github/workflows/test-command.yml b/.github/workflows/test-command.yml index d060a29fc046..f4ea4748ad0f 100644 --- a/.github/workflows/test-command.yml +++ b/.github/workflows/test-command.yml @@ -40,6 +40,7 @@ jobs: integration-test: needs: start-test-runner runs-on: ${{ needs.start-test-runner.outputs.label }} + environment: more-secrets steps: - name: Search for valid connector name format id: regex diff --git a/docs/connector-development/README.md b/docs/connector-development/README.md index 6f025b7115e1..af7a5af0b6a6 100644 --- a/docs/connector-development/README.md +++ b/docs/connector-development/README.md @@ -124,7 +124,7 @@ Once you've finished iterating on the changes to a connector as specified in its In order to run integration tests in CI, you'll often need to inject credentials into CI. There are a few steps for doing this: 1. **Place the credentials into Lastpass**: Airbyte uses a shared Lastpass account as the source of truth for all secrets. Place the credentials **exactly as they should be used by the connector** into a secure note i.e: it should basically be a copy paste of the `config.json` passed into a connector via the `--config` flag. We use the following naming pattern: ` creds` e.g: `source google adwords creds` or `destination snowflake creds`. -2. **Add the credentials to Github Secrets**: To inject credentials into a CI workflow, the first step is to add it to Github Secrets. Admin access to the Airbyte repo is required to do this. All Airbyte engineers have admin access and should be able to do this themselves. External contributors or contractors will need to request this from their team lead or project manager who should have admin access. Follow the same naming pattern as all the other secrets e.g: if you are placing credentials for source google adwords, name the secret `SOURCE_GOOGLE_ADWORDS_CREDS`. After doing this step, the secret will be available in the Github run using the workflow secrets syntax. +2. **Add the credentials to Github Secrets**: To inject credentials into a CI workflow, the first step is to add it to Github Secrets, specifically within the ["more-secrets" environment](https://github.com/airbytehq/airbyte/settings/environments/276695501/edit). Admin access to the Airbyte repo is required to do this. All Airbyte engineers have admin access and should be able to do this themselves. External contributors or contractors will need to request this from their team lead or project manager who should have admin access. Follow the same naming pattern as all the other secrets e.g: if you are placing credentials for source google adwords, name the secret `SOURCE_GOOGLE_ADWORDS_CREDS`. After doing this step, the secret will be available in the relevant Github workflows using the workflow secrets syntax. 3. **Inject the credentials into test and publish CI workflows**: edit the files `.github/workflows/publish-command.yml` and `.github/workflows/test-command.yml` to inject the secret into the CI run. This will make these secrets available to the `/test` and `/publish` commands. 4. **During CI, write the secret from env variables to the connector directory**: edit `tools/bin/ci_credentials.sh` to write the secret into the `secrets/` directory of the relevant connector. 5. That should be it.