Bump CDk #31139

benmoriceau · 2023-10-06T20:29:01Z

What

Bump in order to include the reduction of the async buffer block size

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user?

For connector PRs, use this section to explain which type of semantic versioning bump occurs as a result of the changes. Refer to our Semantic Versioning for Connectors guidelines for more information. Breaking changes to connectors must be documented by an Airbyte engineer (PR author, or reviewer for community PRs) by using the Breaking Change Release Playbook.

If there are breaking changes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Actions

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Connector version is set to 0.0.1
- Dockerfile has version 0.0.1
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
- docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:generateScaffolds then checking in your changes
Documentation which references the generator is updated as needed

Updating the Python CDK

Airbyter

Before merging:

Pull Request description explains what problem it is solving
Code change is unit tested
Build and my-py check pass
Smoke test the change on at least one affected connector
- On Github: Run this workflow, passing --use-local-cdk --name=source-<connector> as options
- Locally: airbyte-ci connectors --use-local-cdk --name=source-<connector> test
PR is reviewed and approved

After merging:

Publish the CDK
- The CDK does not follow proper semantic versioning. Choose minor if this the change has significant user impact or is a breaking change. Choose patch otherwise.
- Write a thoughtful changelog message so we know what was updated.
Merge the platform PR that was auto-created for updating the Connector Builder's CDK version
- This step is optional if the change does not affect the connector builder or declarative connectors.

vercel · 2023-10-06T20:29:07Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Oct 6, 2023 9:13pm

benmoriceau · 2023-10-06T20:29:41Z

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/6436196069
✅ Successfully published Java CDK version=0.1.4!

evantahler · 2023-10-06T20:37:39Z

airbyte-cdk/java/airbyte-cdk/README.md

-| :------ | :--------- | :--------------------------------------------------------- | :------------------------------------ |
+| Version | Date       | Pull Request                                               | Subject                                                                                                                                                        |
+|:--------|:-----------| :--------------------------------------------------------- |:---------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| 0.1.4   | 2023-10-06 | [\#31139](https://github.com/airbytehq/airbyte/pull/31139) | Reduce async buffer                                                                                                                                            |
 | 0.1.1   | 2023-09-28 | [\#30835](https://github.com/airbytehq/airbyte/pull/30835) | JDBC destinations now avoid staging area name collisions by using the raw table name as the stage name. (previously we used the stream name as the stage name) |


Looks like we missed some versions here :/

evantahler · 2023-10-06T20:38:19Z

airbyte-integrations/connectors/destination-redshift/build.gradle

@@ -4,7 +4,7 @@ plugins {
 }


I think we are missing a redshift metadata version bump

I think that the build is not fixed yet so I didn't add it on purpose.

Redshift was not publish with the 30 MB bump because of the build issue

airbyte-oss-build-runner · 2023-10-06T20:40:30Z

destination-snowflake test report (commit `256caa0290`) - ❌

⏲️ Total pipeline duration: 01mn45s

Step	Result
Build connector tar	❌
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

airbyte-oss-build-runner · 2023-10-06T20:42:15Z

destination-bigquery test report (commit `256caa0290`) - ❌

⏲️ Total pipeline duration: 01mn35s

Step	Result
Build connector tar	❌
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

airbyte-oss-build-runner · 2023-10-06T20:43:12Z

destination-redshift test report (commit `256caa0290`) - ❌

⏲️ Total pipeline duration: 46.39s

Step	Result
Build connector tar	❌
Validate metadata for destination-redshift	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

airbyte-oss-build-runner · 2023-10-06T21:10:17Z

destination-bigquery test report (commit `256caa0290`) - ❌

⏲️ Total pipeline duration: 08mn25s

Step	Result
Build connector tar	✅
Build destination-bigquery docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	❌
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

airbyte-oss-build-runner · 2023-10-06T21:26:53Z

destination-snowflake test report (commit `efc926ba17`) - ✅

⏲️ Total pipeline duration: 10mn58s

Step	Result
Build connector tar	✅
Build destination-snowflake docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-snowflake	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

airbyte-oss-build-runner · 2023-10-06T21:29:45Z

destination-redshift test report (commit `efc926ba17`) - ❌

⏲️ Total pipeline duration: 02mn41s

Step	Result
Build connector tar	✅
Build destination-redshift docker image for platform(s) linux/x86_64	❌
Validate metadata for destination-redshift	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

airbyte-oss-build-runner · 2023-10-06T21:37:34Z

destination-bigquery test report (commit `efc926ba17`) - ✅

⏲️ Total pipeline duration: 07mn38s

Step	Result
Build connector tar	✅
Build destination-bigquery docker image for platform(s) linux/x86_64	✅
Java Connector Unit Tests	✅
Java Connector Integration Tests	✅
Validate metadata for destination-bigquery	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

benmoriceau · 2023-10-06T21:49:42Z

/approve-and-merge reason="Redshift is broken"

octavia-approvington · 2023-10-06T21:50:19Z

You did it!

Bump CDk

dee243c

benmoriceau requested a review from edgao October 6, 2023 20:29

octavia-squidington-iii added the CDK Connector Development Kit label Oct 6, 2023

edgao approved these changes Oct 6, 2023

View reviewed changes

Bump CDK and connectors

256caa0

benmoriceau requested a review from a team as a code owner October 6, 2023 20:36

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/bigquery connectors/destination/snowflake connectors/destination/redshift labels Oct 6, 2023

vercel bot deployed to Preview October 6, 2023 20:38 View deployment

evantahler requested changes Oct 6, 2023

View reviewed changes

benmoriceau requested a review from evantahler October 6, 2023 20:40

Fix Unit test

efc926b

octavia-approvington approved these changes Oct 6, 2023

View reviewed changes

octavia-approvington merged commit 9e2b3f4 into master Oct 6, 2023

octavia-approvington deleted the bmoric/bump-cdk branch October 6, 2023 21:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump CDk #31139

Bump CDk #31139

benmoriceau commented Oct 6, 2023

vercel bot commented Oct 6, 2023 •

edited

Loading

benmoriceau commented Oct 6, 2023 •

edited by github-actions bot

Loading

evantahler Oct 6, 2023

benmoriceau Oct 6, 2023

evantahler Oct 6, 2023

benmoriceau Oct 6, 2023

benmoriceau Oct 6, 2023

airbyte-oss-build-runner commented Oct 6, 2023

airbyte-oss-build-runner commented Oct 6, 2023

airbyte-oss-build-runner commented Oct 6, 2023

airbyte-oss-build-runner commented Oct 6, 2023

airbyte-oss-build-runner commented Oct 6, 2023

airbyte-oss-build-runner commented Oct 6, 2023

airbyte-oss-build-runner commented Oct 6, 2023

benmoriceau commented Oct 6, 2023

octavia-approvington commented Oct 6, 2023

Bump CDk #31139

Bump CDk #31139

Conversation

benmoriceau commented Oct 6, 2023

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Airbyter

vercel bot commented Oct 6, 2023 • edited Loading

benmoriceau commented Oct 6, 2023 • edited by github-actions bot Loading

evantahler Oct 6, 2023

Choose a reason for hiding this comment

benmoriceau Oct 6, 2023

Choose a reason for hiding this comment

evantahler Oct 6, 2023

Choose a reason for hiding this comment

benmoriceau Oct 6, 2023

Choose a reason for hiding this comment

benmoriceau Oct 6, 2023

Choose a reason for hiding this comment

airbyte-oss-build-runner commented Oct 6, 2023

destination-snowflake test report (commit 256caa0290) - ❌

airbyte-oss-build-runner commented Oct 6, 2023

destination-bigquery test report (commit 256caa0290) - ❌

airbyte-oss-build-runner commented Oct 6, 2023

destination-redshift test report (commit 256caa0290) - ❌

airbyte-oss-build-runner commented Oct 6, 2023

destination-bigquery test report (commit 256caa0290) - ❌

airbyte-oss-build-runner commented Oct 6, 2023

destination-snowflake test report (commit efc926ba17) - ✅

airbyte-oss-build-runner commented Oct 6, 2023

destination-redshift test report (commit efc926ba17) - ❌

airbyte-oss-build-runner commented Oct 6, 2023

destination-bigquery test report (commit efc926ba17) - ✅

benmoriceau commented Oct 6, 2023

octavia-approvington commented Oct 6, 2023

vercel bot commented Oct 6, 2023 •

edited

Loading

benmoriceau commented Oct 6, 2023 •

edited by github-actions bot

Loading

destination-snowflake test report (commit `256caa0290`) - ❌

destination-bigquery test report (commit `256caa0290`) - ❌

destination-redshift test report (commit `256caa0290`) - ❌

destination-bigquery test report (commit `256caa0290`) - ❌

destination-snowflake test report (commit `efc926ba17`) - ✅

destination-redshift test report (commit `efc926ba17`) - ❌

destination-bigquery test report (commit `efc926ba17`) - ✅