Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bump testcontainers version to latest #30493

Merged
merged 2 commits into from
Sep 18, 2023

Conversation

postamar
Copy link
Contributor

@postamar postamar commented Sep 15, 2023

This PR bumps all testcontainers package versions to the latest across the board, following up on work started by @alafanechere. We need to do this to have access to images for the architectures that we use (arm machines on aws, apple silicon locally...). The need for this was especially pressing for source-mysql apparently, airbyte-ci just doesn't work for that connector right now.

Notice that I tie all testcontainers dependencies to one version number. This is deliberate, any testcontainers package will transitively depend on the testcontainers base package of the same version. To use a specific, different version in a connector for whatever reason, let's import it using a hardcoded string instead of a reference to deps.toml. @aaronsteers perhaps the CDK will formalize things in this respect? It seems sensible that testcontainers version pinning could happen in there.

I made a reasonable effort to test that this doesn't break anything that I think that we care about. This change actually breaks many connector's tests, but I haven't worried about those which were already routinely broken according to https://connectors.airbyte.com/files/generated_reports/connector_registry_report.html. The fixes, in any case, are fairly straightforward.

This PR will trigger a bunch of CI runs, which is actually desirable for a change. I'll go over the reports after the week-end, once they've had a chance to run. I did some manual tests before publishing this that verified that this fixes the source-mysql + strict-encrypt builds so this is ready for review. If any other connectors need fixing the changes are along the same lines so that shouldn't block the review.

Informs #29463

@github-actions
Copy link
Contributor

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

@github-actions
Copy link
Contributor

destination-cassandra test report (commit 329ec73179) - ❌

⏲️ Total pipeline duration: 07mn24s

Step Result
Build connector tar
Build destination-cassandra docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Validate airbyte-integrations/connectors/destination-cassandra/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-cassandra test

@github-actions
Copy link
Contributor

source-mysql test report (commit 329ec73179) - ✅

⏲️ Total pipeline duration: 02mn42s

Step Result
Build connector tar
Build source-mysql docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Acceptance tests
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

@github-actions
Copy link
Contributor

destination-elasticsearch-strict-encrypt test report (commit 329ec73179) - ❌

⏲️ Total pipeline duration: 03mn44s

Step Result
Build connector tar
Build destination-elasticsearch-strict-encrypt docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Validate airbyte-integrations/connectors/destination-elasticsearch-strict-encrypt/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-elasticsearch-strict-encrypt test

@github-actions
Copy link
Contributor

destination-mongodb-strict-encrypt test report (commit 329ec73179) - ✅

⏲️ Total pipeline duration: 03mn12s

Step Result
Build connector tar
Build destination-mongodb-strict-encrypt docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Validate airbyte-integrations/connectors/destination-mongodb-strict-encrypt/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mongodb-strict-encrypt test

@github-actions
Copy link
Contributor

Coverage report for source-postgres

There is no coverage information present for the Files changed

Total Project Coverage 70.6% 🍏

@github-actions
Copy link
Contributor

destination-redis test report (commit 329ec73179) - ❌

⏲️ Total pipeline duration: 04mn45s

Step Result
Build connector tar
Build destination-redis docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Validate airbyte-integrations/connectors/destination-redis/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redis test

Copy link
Collaborator

@aaronsteers aaronsteers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense! If you do note any new failures, and if you can document those for reference, that is much appreciated. That said, I now it is tricky to identify 'new' failures in cases where tests are failing already, so I wouldn't block on it either way.

@github-actions
Copy link
Contributor

source-clickhouse-strict-encrypt test report (commit 329ec73179) - ❌

⏲️ Total pipeline duration: 302mn11s

Step Result
Build connector tar
Build source-clickhouse-strict-encrypt docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Acceptance tests
Validate airbyte-integrations/connectors/source-clickhouse-strict-encrypt/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-clickhouse-strict-encrypt test

@github-actions
Copy link
Contributor

source-postgres test report (commit 329ec73179) - ✅

⏲️ Total pipeline duration: 23mn52s

Step Result
Build connector tar
Build source-postgres docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Acceptance tests
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

@github-actions
Copy link
Contributor

source-mssql-strict-encrypt test report (commit 329ec73179) - ❌

⏲️ Total pipeline duration: 311mn41s

Step Result
Build connector tar
Build source-mssql-strict-encrypt docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Acceptance tests
Validate airbyte-integrations/connectors/source-mssql-strict-encrypt/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mssql-strict-encrypt test

@github-actions
Copy link
Contributor

destination-postgres-strict-encrypt test report (commit 329ec73179) - ❌

⏲️ Total pipeline duration: 09mn13s

Step Result
Build connector tar
Build destination-postgres-strict-encrypt docker image for platform linux/x86_64
Java Connector Unit Tests
Build airbyte/normalization:dev
Java Connector Integration Tests
Validate airbyte-integrations/connectors/destination-postgres-strict-encrypt/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-postgres-strict-encrypt test

@github-actions
Copy link
Contributor

destination-pulsar test report (commit 329ec73179) - ❌

⏲️ Total pipeline duration: 34mn54s

Step Result
Build connector tar
Build destination-pulsar docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Validate airbyte-integrations/connectors/destination-pulsar/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-pulsar test

@github-actions
Copy link
Contributor

source-cockroachdb-strict-encrypt test report (commit 329ec73179) - ❌

⏲️ Total pipeline duration: 50.25s

Step Result
Build connector tar
Validate airbyte-integrations/connectors/source-cockroachdb-strict-encrypt/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-cockroachdb-strict-encrypt test

@github-actions
Copy link
Contributor

destination-kafka test report (commit 329ec73179) - ❌

⏲️ Total pipeline duration: 06mn35s

Step Result
Build connector tar
Build destination-kafka docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Validate airbyte-integrations/connectors/destination-kafka/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-kafka test

@github-actions
Copy link
Contributor

destination-mssql-strict-encrypt test report (commit 329ec73179) - ❌

⏲️ Total pipeline duration: 01mn43s

Step Result
Build connector tar
Build destination-mssql-strict-encrypt docker image for platform linux/x86_64
Validate airbyte-integrations/connectors/destination-mssql-strict-encrypt/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mssql-strict-encrypt test

@github-actions
Copy link
Contributor

source-db2 test report (commit 329ec73179) - ❌

⏲️ Total pipeline duration: 03mn41s

Step Result
Build connector tar
Build source-db2 docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Acceptance tests
Validate airbyte-integrations/connectors/source-db2/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-db2 test

@github-actions
Copy link
Contributor

destination-mongodb test report (commit 329ec73179) - ❌

⏲️ Total pipeline duration: 13mn37s

Step Result
Build connector tar
Build destination-mongodb docker image for platform linux/x86_64
Java Connector Unit Tests
Java Connector Integration Tests
Validate airbyte-integrations/connectors/destination-mongodb/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-mongodb test

@github-actions
Copy link
Contributor

destination-oracle test report (commit 329ec73179) - ❌

⏲️ Total pipeline duration: 302mn44s

Step Result
Build connector tar
Build destination-oracle docker image for platform linux/x86_64
Java Connector Unit Tests
Build airbyte/normalization-oracle:dev
Java Connector Integration Tests
Validate airbyte-integrations/connectors/destination-oracle/metadata.yaml
Connector version semver check
QA checks

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-oracle test

@postamar
Copy link
Contributor Author

FYI @airbytehq/connector-operations the CI hit some github rate limiter:

github.GithubException.RateLimitExceededException: 403 ***"message": "API rate limit exceeded for installation ID 6913263. If you reach out to GitHub Support for help, please include the request ID B91A:17CF:2795E19:28DB178:6505A23C.", "documentation_url": "https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"***

I can't imagine any scenario where this is desirable. It's bad enough that the whole thing takes several days, frankly.

I'm going to merge this PR now and deal with the consequences later.

@postamar
Copy link
Contributor Author

/approve-and-merge reason="need to unblock source-mysql"

@octavia-approvington
Copy link
Contributor

Jerry would be proud
Spongebob

@octavia-approvington octavia-approvington merged commit a5eb24e into master Sep 18, 2023
@octavia-approvington octavia-approvington deleted the postamar/bump-testcontainers-version branch September 18, 2023 08:50
@postamar postamar linked an issue Sep 20, 2023 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues connectors/destination/cassandra connectors/destination/clickhouse connectors/destination/clickhouse-strict-encrypt connectors/destination/elasticsearch connectors/destination/elasticsearch-strict-encryp connectors/destination/iceberg connectors/destination/kafka connectors/destination/mariadb-columnstore connectors/destination/mongodb connectors/destination/mongodb-strict-encrypt connectors/destination/mssql connectors/destination/mssql-strict-encrypt connectors/destination/mysql connectors/destination/mysql-strict-encrypt connectors/destination/oracle connectors/destination/oracle-strict-encrypt connectors/destination/postgres connectors/destination/postgres-strict-encrypt connectors/destination/pulsar connectors/destination/redis connectors/destination/scylla connectors/destination/tidb connectors/source/alloydb connectors/source/alloydb-strict-encrypt connectors/source/clickhouse connectors/source/clickhouse-strict-encrypt connectors/source/cockroachdb connectors/source/cockroachdb-strict-encrypt connectors/source/db2-strict-encrypt connectors/source/db2 connectors/source/elasticsearch connectors/source/jdbc connectors/source/kafka connectors/source/mongodb-v2 connectors/source/mssql connectors/source/mssql-strict-encrypt connectors/source/mysql connectors/source/mysql-strict-encrypt connectors/source/oracle connectors/source/oracle-strict-encrypt connectors/source/postgres connectors/source/postgres-strict-encrypt connectors/source/relational-db connectors/source/sftp connectors/source/tidb
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bump testcontainers version to latest
5 participants