Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Snowflake Destination: use better file size with S3 staging files #9920

Merged
merged 21 commits into from
Feb 15, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
6eb7333
split s3 staging files to files by 100 Mb and removed legacyS3StreamC…
andriikorotkov Jan 31, 2022
bebeef8
split s3 staging files to files by 100 Mb and removed legacyS3StreamC…
andriikorotkov Jan 31, 2022
c3f4dc8
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Jan 31, 2022
1811359
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Feb 1, 2022
6dd1c4b
updated code style
andriikorotkov Feb 1, 2022
d764804
fix remarks
andriikorotkov Feb 2, 2022
1641f46
fix remarks
andriikorotkov Feb 2, 2022
3569bfc
fix code style
andriikorotkov Feb 3, 2022
2e4480b
fix remarks
andriikorotkov Feb 3, 2022
9890715
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Feb 3, 2022
1f2bc7d
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Feb 7, 2022
a6004f4
fix remarks
andriikorotkov Feb 7, 2022
ff04702
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Feb 8, 2022
18a915f
fix remarks
andriikorotkov Feb 8, 2022
219f59f
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Feb 8, 2022
644d4f5
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Feb 9, 2022
b54ab2f
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Feb 11, 2022
2991552
updated documentations and images versions
andriikorotkov Feb 11, 2022
cb6db6e
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Feb 14, 2022
956f066
Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…
andriikorotkov Feb 15, 2022
08840f9
updated documentation
andriikorotkov Feb 15, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
updated documentations and images versions
  • Loading branch information
andriikorotkov committed Feb 11, 2022
commit 29915522e097432c3c215cdefccf32a817dde7f5
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"destinationDefinitionId": "424892c4-daac-4491-b35d-c6688ba547ba",
"name": "Snowflake",
"dockerRepository": "airbyte/destination-snowflake",
"dockerImageTag": "0.4.6",
"dockerImageTag": "0.4.9",
"documentationUrl": "https://docs.airbyte.io/integrations/destinations/snowflake",
"icon": "snowflake.svg"
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"destinationDefinitionId": "f7a7d195-377f-cf5b-70a5-be6b819019dc",
"name": "Redshift",
"dockerRepository": "airbyte/destination-redshift",
"dockerImageTag": "0.3.19",
"dockerImageTag": "0.3.24",
"documentationUrl": "https://docs.airbyte.io/integrations/destinations/redshift",
"icon": "redshift.svg"
}
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@
- name: Redshift
destinationDefinitionId: f7a7d195-377f-cf5b-70a5-be6b819019dc
dockerRepository: airbyte/destination-redshift
dockerImageTag: 0.3.23
dockerImageTag: 0.3.24
documentationUrl: https://docs.airbyte.io/integrations/destinations/redshift
icon: redshift.svg
- name: Rockset
Expand All @@ -185,7 +185,7 @@
- name: Snowflake
destinationDefinitionId: 424892c4-daac-4491-b35d-c6688ba547ba
dockerRepository: airbyte/destination-snowflake
dockerImageTag: 0.4.8
dockerImageTag: 0.4.9
documentationUrl: https://docs.airbyte.io/integrations/destinations/snowflake
icon: snowflake.svg
- name: MariaDB ColumnStore
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3264,7 +3264,7 @@
supported_destination_sync_modes:
- "overwrite"
- "append"
- dockerImage: "airbyte/destination-redshift:0.3.23"
- dockerImage: "airbyte/destination-redshift:0.3.24"
spec:
documentationUrl: "https://docs.airbyte.io/integrations/destinations/redshift"
connectionSpecification:
Expand Down Expand Up @@ -3817,7 +3817,7 @@
supported_destination_sync_modes:
- "overwrite"
- "append"
- dockerImage: "airbyte/destination-snowflake:0.4.8"
- dockerImage: "airbyte/destination-snowflake:0.4.9"
spec:
documentationUrl: "https://docs.airbyte.io/integrations/destinations/snowflake"
connectionSpecification:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-jdbc

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.3.8
LABEL io.airbyte.version=0.3.9
LABEL io.airbyte.name=airbyte/destination-jdbc
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ ENV APPLICATION destination-redshift

COPY --from=build /airbyte /airbyte

LABEL io.airbyte.version=0.3.23
LABEL io.airbyte.version=0.3.24
LABEL io.airbyte.name=airbyte/destination-redshift
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar

RUN tar xf ${APPLICATION}.tar --strip-components=1

ENV APPLICATION_VERSION 0.4.8
ENV APPLICATION_VERSION 0.4.9
ENV ENABLE_SENTRY true

LABEL io.airbyte.version=0.4.8
LABEL io.airbyte.version=0.4.9
LABEL io.airbyte.name=airbyte/destination-snowflake
1 change: 1 addition & 0 deletions docs/integrations/destinations/redshift.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ All Redshift connections are encrypted using SSL

| Version | Date | Pull Request | Subject |
| :------ | :-------- | :----- | :------ |
| 0.3.24 | 2022-02-11 | [#9920](https://github.com/airbytehq/airbyte/pull/9920) | Updated the size of staging files for S3 staging. Also, added closure of S3 writers to staging files when data has been written to an staging file. |
| 0.3.23 | 2021-12-16 | [\#8855](https://github.com/airbytehq/airbyte/pull/8855) | Add `purgeStagingData` option to enable/disable deleting the staging data |
| 0.3.22 | 2021-12-15 | [#8607](https://github.com/airbytehq/airbyte/pull/8607) | Accept a path for the staging data |
| 0.3.21 | 2021-12-10 | [#8562](https://github.com/airbytehq/airbyte/pull/8562) | Moving classes around for better dependency management |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/snowflake.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,7 @@ Finally, you need to add read/write permissions to your bucket with that email.

| Version | Date | Pull Request | Subject |
|:--------|:-----------| :----- | :------ |
| 0.4.9 | 2022-02-11 | [#9920](https://github.com/airbytehq/airbyte/pull/9920) | Updated the size of staging files for S3 staging. Also, added closure of S3 writers to staging files when data has been written to an staging file. |
| 0.4.8 | 2022-02-01 | [\#9959](https://github.com/airbytehq/airbyte/pull/9959) | Fix null pointer exception from buffered stream consumer. |
| 0.4.7 | 2022-01-29 | [\#9745](https://github.com/airbytehq/airbyte/pull/9745) | Integrate with Sentry. |
| 0.4.6 | 2022-01-28 | [#9623](https://github.com/airbytehq/airbyte/pull/9623) | Add jdbc_url_params support for optional JDBC parameters |
Expand Down