🎉 Destination-snowflake: start using new S3StreamCopier, and expose the purgeStagingData option #9531

etsybaev · 2022-01-16T14:36:01Z

What

Currently, Snowflake uses Deprecated own s3 writer

How

Updated connector to use central s3 writer in the same way like we already have for Redshift connector

Selected "dev" image with changes to use:

2 streams selected to migrate from test Postgres source to Snoflake:
Tested on UI locally

Result on Snowflake:

🚨 User Impact 🚨

PurgeStagingData option is added

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

…ose the purgeStagingData option

…-stream-copier # Conflicts: # airbyte-integrations/connectors/destination-snowflake/src/test/java/io/airbyte/integrations/destination/snowflake/SnowflakeDestinationTest.java

.../src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeCopyS3Destination.java

.../main/java/io/airbyte/integrations/destination/snowflake/SnowflakeS3StreamCopierFactory.java

etsybaev · 2022-01-17T14:27:33Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1708438468
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1708438468
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     155      8    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 520    333    36%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     34    77%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1258    525    58%

...ake/src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeS3StreamCopier.java

.../src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeCopyS3Destination.java

...src/test/java/io/airbyte/integrations/destination/snowflake/SnowflakeS3StreamCopierTest.java

tuliren · 2022-01-19T01:00:11Z

airbyte-integrations/connectors/destination-snowflake/README.md

@@ -22,7 +22,7 @@
 Put the contents of the `Snowflake Integration Test Config` secret on Rippling under the `Engineering` folder into `secrets/config.json` to be able to run integration tests locally.

 1. Put the contents of the `destination snowflake - insert test creds` LastPass secret into `secrets/insert_config.json`.
-1. Put the contents of the `destination snowflake - insert staging test creds` secret into `insert_staging_config.json`.
+1. Put the contents of the `destination snowflake - insert staging test creds` secret into `internal_staging_config.json`.


Should this be secrets/internal_staging_config?

Have no idea :) I just aligned documentation to what we actually have in codebase and secret manager

…-stream-copier

etsybaev · 2022-01-19T17:46:56Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1719424923
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1719424923
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     155      8    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 520    333    36%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     34    77%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1258    525    58%

etsybaev · 2022-01-19T17:47:30Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1719427262
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1719427262
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     155      8    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 520    333    36%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     34    77%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1258    525    58%

…-stream-copier

edgao

nice!

…-stream-copier

etsybaev · 2022-01-20T17:20:48Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1724681686
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1724681686
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     155      8    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 520    333    36%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     34    77%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1258    525    58%

etsybaev · 2022-01-20T17:21:13Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1724683120
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1724683120
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                              Stmts   Miss  Cover
	 -------------------------------------------------------------------------------------
	 main_dev_transform_catalog.py                                         3      3     0%
	 main_dev_transform_config.py                                          3      3     0%
	 normalization/__init__.py                                             4      0   100%
	 normalization/destination_type.py                                    13      0   100%
	 normalization/transform_catalog/__init__.py                           2      0   100%
	 normalization/transform_catalog/catalog_processor.py                143     77    46%
	 normalization/transform_catalog/destination_name_transformer.py     155      8    95%
	 normalization/transform_catalog/reserved_keywords.py                 13      0   100%
	 normalization/transform_catalog/stream_processor.py                 520    333    36%
	 normalization/transform_catalog/table_name_registry.py              174     34    80%
	 normalization/transform_catalog/transform.py                         45     26    42%
	 normalization/transform_catalog/utils.py                             33      7    79%
	 normalization/transform_config/__init__.py                            2      0   100%
	 normalization/transform_config/transform.py                         148     34    77%
	 -------------------------------------------------------------------------------------
	 TOTAL                                                              1258    525    58%

etsybaev · 2022-01-20T18:47:12Z

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1725040222
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1725040222

[8820] Destination-snowflake: start using new S3StreamCopier, and exp…

dfb0d46

…ose the purgeStagingData option

etsybaev linked an issue Jan 16, 2022 that may be closed by this pull request

Snowflake S3 copy destination should use the updated S3StreamCopier, and expose the purgeStagingData option #8820

Closed

github-actions bot added the area/connectors Connector related issues label Jan 16, 2022

etsybaev temporarily deployed to more-secrets January 16, 2022 14:37 Inactive

Merge branch 'master' into etsybaev/8820-destination-snowlake-use-new…

26b7cac

…-stream-copier # Conflicts: # airbyte-integrations/connectors/destination-snowflake/src/test/java/io/airbyte/integrations/destination/snowflake/SnowflakeDestinationTest.java

etsybaev temporarily deployed to more-secrets January 16, 2022 16:51 Inactive

etsybaev requested a review from mkhokh-33 January 17, 2022 08:33

mkhokh-33 suggested changes Jan 17, 2022

View reviewed changes

.../src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeCopyS3Destination.java Outdated Show resolved Hide resolved

.../main/java/io/airbyte/integrations/destination/snowflake/SnowflakeS3StreamCopierFactory.java Outdated Show resolved Hide resolved

Fixed code as per comments in PR

66d5f1b

etsybaev requested a review from mkhokh-33 January 17, 2022 14:26

etsybaev temporarily deployed to more-secrets January 17, 2022 14:28 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 17, 2022 14:29 Inactive

mkhokh-33 approved these changes Jan 17, 2022

View reviewed changes

karinakuz added connectors/destinations-warehouse connectors/destination/snowflake labels Jan 17, 2022

etsybaev changed the title ~~[DRAFT_PR_IGNORE_IT][8820] Destination-snowflake: start using new S3StreamCopier, and exp…~~ 🎉 Destination-snowflake: start using new S3StreamCopier, and expose the purgeStagingData option Jan 18, 2022

etsybaev marked this pull request as ready for review January 18, 2022 10:50

alexandr-shegeda requested a review from edgao January 18, 2022 14:17

etsybaev requested review from tuliren and removed request for edgao January 18, 2022 16:13

edgao requested changes Jan 18, 2022

View reviewed changes

tuliren approved these changes Jan 19, 2022

View reviewed changes

etsybaev added 2 commits January 19, 2022 15:20

Merge branch 'master' into etsybaev/8820-destination-snowlake-use-new…

087a7a4

…-stream-copier

Fixed code as per comments in PR(2)

ee4a894

etsybaev temporarily deployed to more-secrets January 19, 2022 16:26 Inactive

minor refactor + fix code style

010f36f

etsybaev temporarily deployed to more-secrets January 19, 2022 17:30 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 19, 2022 17:49 Inactive

etsybaev added 2 commits January 20, 2022 18:26

Extended test to check all staging files

e4abe14

Merge branch 'master' into etsybaev/8820-destination-snowlake-use-new…

715a1a0

…-stream-copier

etsybaev temporarily deployed to more-secrets January 20, 2022 16:28 Inactive

etsybaev requested a review from edgao January 20, 2022 16:29

edgao mentioned this pull request Jan 20, 2022

🎉 Detect changes of CDK and SAT folders for SonarQube #9612

Closed

edgao approved these changes Jan 20, 2022

View reviewed changes

Merge branch 'master' into etsybaev/8820-destination-snowlake-use-new…

a83204a

…-stream-copier

etsybaev temporarily deployed to more-secrets January 20, 2022 17:21 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 20, 2022 17:23 Inactive

Updated documentation and bumped version

45323cb

github-actions bot added the area/documentation Improvements or additions to documentation label Jan 20, 2022

etsybaev temporarily deployed to more-secrets January 20, 2022 18:47 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 20, 2022 18:49 Inactive

Updated destination-spec

e00aa92

etsybaev merged commit 9fe804a into master Jan 20, 2022

etsybaev deleted the etsybaev/8820-destination-snowlake-use-new-stream-copier branch January 20, 2022 20:21

etsybaev temporarily deployed to more-secrets January 20, 2022 20:23 Inactive

octavia-squidington-iii mentioned this pull request Jan 21, 2022

Bump Airbyte version from 0.35.6-alpha to 0.35.7-alpha #9708

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Destination-snowflake: start using new S3StreamCopier, and expose the purgeStagingData option #9531

🎉 Destination-snowflake: start using new S3StreamCopier, and expose the purgeStagingData option #9531

etsybaev commented Jan 16, 2022 •

edited

Loading

etsybaev commented Jan 17, 2022 •

edited by github-actions bot

Loading

tuliren Jan 19, 2022

etsybaev Jan 19, 2022 •

edited

Loading

etsybaev commented Jan 19, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 19, 2022 •

edited by github-actions bot

Loading

edgao left a comment

etsybaev commented Jan 20, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 20, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 20, 2022 •

edited by github-actions bot

Loading

🎉 Destination-snowflake: start using new S3StreamCopier, and expose the purgeStagingData option #9531

🎉 Destination-snowflake: start using new S3StreamCopier, and expose the purgeStagingData option #9531

Conversation

etsybaev commented Jan 16, 2022 • edited Loading

What

How

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

etsybaev commented Jan 17, 2022 • edited by github-actions bot Loading

tuliren Jan 19, 2022

Choose a reason for hiding this comment

etsybaev Jan 19, 2022 • edited Loading

Choose a reason for hiding this comment

etsybaev commented Jan 19, 2022 • edited by github-actions bot Loading

etsybaev commented Jan 19, 2022 • edited by github-actions bot Loading

edgao left a comment

Choose a reason for hiding this comment

etsybaev commented Jan 20, 2022 • edited by github-actions bot Loading

etsybaev commented Jan 20, 2022 • edited by github-actions bot Loading

etsybaev commented Jan 20, 2022 • edited by github-actions bot Loading

etsybaev commented Jan 16, 2022 •

edited

Loading

etsybaev commented Jan 17, 2022 •

edited by github-actions bot

Loading

etsybaev Jan 19, 2022 •

edited

Loading

etsybaev commented Jan 19, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 19, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 20, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 20, 2022 •

edited by github-actions bot

Loading

etsybaev commented Jan 20, 2022 •

edited by github-actions bot

Loading