Skip to content

Commit

Permalink
Temporary solution for hitting secrets limit (#5085)
Browse files Browse the repository at this point in the history
* add more-secrets environment to publish and test workflows

* add link to more-secrets environment in relevant docs location
  • Loading branch information
Phlair authored Jul 29, 2021
1 parent 0d2364c commit 2fe0043
Show file tree
Hide file tree
Showing 3 changed files with 3 additions and 1 deletion.
1 change: 1 addition & 0 deletions .github/workflows/publish-command.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ jobs:
publish-image:
needs: start-publish-image-runner
runs-on: ${{ needs.start-publish-image-runner.outputs.label }}
environment: more-secrets
steps:
- name: Search for valid connector name format
id: regex
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/test-command.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ jobs:
integration-test:
needs: start-test-runner
runs-on: ${{ needs.start-test-runner.outputs.label }}
environment: more-secrets
steps:
- name: Search for valid connector name format
id: regex
Expand Down
2 changes: 1 addition & 1 deletion docs/connector-development/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ Once you've finished iterating on the changes to a connector as specified in its
In order to run integration tests in CI, you'll often need to inject credentials into CI. There are a few steps for doing this:

1. **Place the credentials into Lastpass**: Airbyte uses a shared Lastpass account as the source of truth for all secrets. Place the credentials **exactly as they should be used by the connector** into a secure note i.e: it should basically be a copy paste of the `config.json` passed into a connector via the `--config` flag. We use the following naming pattern: `<source OR destination> <name> creds` e.g: `source google adwords creds` or `destination snowflake creds`.
2. **Add the credentials to Github Secrets**: To inject credentials into a CI workflow, the first step is to add it to Github Secrets. Admin access to the Airbyte repo is required to do this. All Airbyte engineers have admin access and should be able to do this themselves. External contributors or contractors will need to request this from their team lead or project manager who should have admin access. Follow the same naming pattern as all the other secrets e.g: if you are placing credentials for source google adwords, name the secret `SOURCE_GOOGLE_ADWORDS_CREDS`. After doing this step, the secret will be available in the Github run using the workflow secrets syntax.
2. **Add the credentials to Github Secrets**: To inject credentials into a CI workflow, the first step is to add it to Github Secrets, specifically within the ["more-secrets" environment](https://github.com/airbytehq/airbyte/settings/environments/276695501/edit). Admin access to the Airbyte repo is required to do this. All Airbyte engineers have admin access and should be able to do this themselves. External contributors or contractors will need to request this from their team lead or project manager who should have admin access. Follow the same naming pattern as all the other secrets e.g: if you are placing credentials for source google adwords, name the secret `SOURCE_GOOGLE_ADWORDS_CREDS`. After doing this step, the secret will be available in the relevant Github workflows using the workflow secrets syntax.
3. **Inject the credentials into test and publish CI workflows**: edit the files `.github/workflows/publish-command.yml` and `.github/workflows/test-command.yml` to inject the secret into the CI run. This will make these secrets available to the `/test` and `/publish` commands.
4. **During CI, write the secret from env variables to the connector directory**: edit `tools/bin/ci_credentials.sh` to write the secret into the `secrets/` directory of the relevant connector.
5. That should be it.

0 comments on commit 2fe0043

Please sign in to comment.