🐛 Source Amplitude: fix datetime update and session length change #28942

maxi297 · 2023-08-01T20:08:50Z

What

Fixing p1-amplitude-state-format-issue (maybe escalate to p0)

How

Support many datetime format for cursor
Update extraction on session length

Breaking change document: https://docs.google.com/document/d/1thRH0nvccNgtesxV8ia3QolwH1t1Lz2yanRWbfr3VAI/edit

github-actions · 2023-08-01T20:09:12Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

maxi297 · 2023-08-01T20:10:41Z

/legacy-test connector=connectors/source-amplitude local_cdk=1

🕑 connectors/source-amplitude https://github.com/airbytehq/airbyte/actions/runs/5731381621
❌ connectors/source-amplitude https://github.com/airbytehq/airbyte/actions/runs/5731381621
🐛 https://gradle.com/s/ubdigkp3gu46k

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Please check...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:105: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:628: The previous and actual discovered catalogs are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:725: This tests currently leads to too much failures. We need to fix the connectors at scale first.
======= 1 failed, 38 passed, 3 skipped, 1 warning in 2446.29s (0:40:46) ========

octavia-squidington-iii · 2023-08-01T20:13:26Z

source-amplitude test report (commit `689657f37f`) - ❌

⏲️ Total pipeline duration: 02mn19s

Step	Result
Validate airbyte-integrations/connectors/source-amplitude/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Code format checks	✅
Connector package install	✅
Build source-amplitude docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-amplitude test

octavia-squidington-iii · 2023-08-01T20:19:18Z

source-amplitude test report (commit `ff44713964`) - ❌

⏲️ Total pipeline duration: 02mn18s

Step	Result
Validate airbyte-integrations/connectors/source-amplitude/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Code format checks	✅
Connector package install	✅
Build source-amplitude docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-amplitude test

octavia-squidington-iii · 2023-08-01T22:04:44Z

source-amplitude test report (commit `fb3e464fc0`) - ❌

⏲️ Total pipeline duration: 02mn50s

Step	Result
Validate airbyte-integrations/connectors/source-amplitude/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-amplitude docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-amplitude test

…e-formats

maxi297 · 2023-08-01T22:09:01Z

airbyte-cdk/python/unit_tests/sources/declarative/incremental/test_datetime_based_cursor.py

@@ -618,7 +618,7 @@ def test_cursor_granularity_but_no_step():
        )


-def test_given_multiple_cursor_datetime_format_then_slice_using_first_format():
+def test_given_multiple_cursor_datetime_format_then_slice_using_datetime_format():


Just cleaning up naming but doesn't affect this PR

octavia-squidington-iii · 2023-08-01T22:31:31Z

source-amplitude test report (commit `a92246ef04`) - ❌

⏲️ Total pipeline duration: 02mn05s

Step	Result
Validate airbyte-integrations/connectors/source-amplitude/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Code format checks	✅
Connector package install	✅
Build source-amplitude docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-amplitude test

octavia-squidington-iii · 2023-08-01T22:44:34Z

source-amplitude test report (commit `b9e613b16b`) - ❌

⏲️ Total pipeline duration: 02mn21s

Step	Result
Validate airbyte-integrations/connectors/source-amplitude/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Code format checks	✅
Connector package install	✅
Build source-amplitude docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-amplitude test

octavia-squidington-iii · 2023-08-01T22:54:02Z

source-amplitude test report (commit `eb9ff2b7e6`) - ❌

⏲️ Total pipeline duration: 03mn15s

Step	Result
Validate airbyte-integrations/connectors/source-amplitude/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Code format checks	✅
Connector package install	✅
Build source-amplitude docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-amplitude test

brianjlai · 2023-08-01T23:10:33Z

airbyte-integrations/connectors/source-amplitude/source_amplitude/components.py

@@ -34,7 +34,7 @@ def extract_records(self, response: requests.Response) -> List[Record]:
            series = response_data.get("series", [])
            if len(series) > 0:
                series = series[0]  # get the nested list
-                return [{"date": date, "length": length} for date, length in zip(response_data["xValues"], series)]
+                return [{"date": date, "length": length["value"]} for date, length in zip(response_data["xValues"], series)]


Why doesn't the change from using length to the length's value key result in a change to expected records? I'm a little confused where this value key comes from based on the API docs: https://www.docs.developers.amplitude.com/analytics/apis/dashboard-rest-api/#response_3

Synced with Maxime, Amplitude response has also changed (but not documented) that has changed what was original an array of numbers:

[12.0, 15.0]

to an array of objects:

[{ "setId": "", "value": 12.0 }, { "setId": "", "value": 15.0 }]

We need this to retain compatibility with our existing connector behavior so this is all good.

…e-formats

brianjlai

ship it! (after we vet the breaking changes with TCS)

octavia-squidington-iii · 2023-08-01T23:34:08Z

source-amplitude test report (commit `9ed1c6defc`) - ❌

⏲️ Total pipeline duration: 37mn42s

Step	Result
Validate airbyte-integrations/connectors/source-amplitude/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Code format checks	✅
Connector package install	✅
Build source-amplitude docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-amplitude test

girarda

lgtm 🚢

octavia-squidington-iii · 2023-08-02T00:42:41Z

source-amplitude test report (commit `425a7da08d`) - ❌

⏲️ Total pipeline duration: 40mn39s

Step	Result
Validate airbyte-integrations/connectors/source-amplitude/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Code format checks	✅
Connector package install	✅
Build source-amplitude docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-amplitude test

…e-formats

octavia-squidington-iii · 2023-08-02T14:01:56Z

source-amplitude test report (commit `7e25160417`) - ❌

⏲️ Total pipeline duration: 02mn34s

Step	Result
Validate airbyte-integrations/connectors/source-amplitude/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Code format checks	✅
Connector package install	✅
Build source-amplitude docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-amplitude test

…e-formats

octavia-squidington-iii · 2023-08-02T20:44:01Z

source-amplitude test report (commit `e072a2fa1e`) - ❌

⏲️ Total pipeline duration: 02mn28s

Step	Result
Validate airbyte-integrations/connectors/source-amplitude/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Code format checks	✅
Connector package install	✅
Build source-amplitude docker image for platform linux/x86_64	✅
Unit tests	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-amplitude test

maxi297 added 2 commits August 1, 2023 15:36

[ISSUE #28782] support multiple cursor field datetime formats

9c3e2c1

Fix amplitude datetime update and session length change

689657f

octavia-squidington-iii added area/connectors Connector related issues connectors/source/amplitude labels Aug 1, 2023

maxi297 changed the title ~~Fix amplitude datetime update and session length change~~ 🐛 Source Amplitude: fix datetime update and session length change Aug 1, 2023

bump version and changelogs

ff44713

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Aug 1, 2023

maxi297 added checklist-action-run and removed checklist-action-run labels Aug 1, 2023

Update schema

fb3e464

Base automatically changed from issue-28782/add-many-cursor-datetime-formats to master August 1, 2023 21:59

maxi297 requested a review from a team as a code owner August 1, 2023 21:59

maxi297 changed the base branch from master to 12255/support-key-pair-auth-for-snowflake-dest August 1, 2023 22:03

maxi297 requested a review from a team as a code owner August 1, 2023 22:03

maxi297 changed the base branch from 12255/support-key-pair-auth-for-snowflake-dest to master August 1, 2023 22:03

Merge branch 'master' into maxi297/p0-source-amplitude-cursor-datetim…

953bc45

…e-formats

octavia-squidington-iii added the CDK Connector Development Kit label Aug 1, 2023

maxi297 commented Aug 1, 2023

View reviewed changes

Automated Commit - Formatting Changes

1fe733e

maxi297 requested review from lazebnyi and removed request for a team August 1, 2023 22:17

Bump minor as this is a breaking change now

a92246e

maxi297 added the breaking-change Don't merge me unless you are ready. label Aug 1, 2023

Retrigger pipeline

b9e613b

Force CDK version

eb9ff2b

Retrigger pipeline

9ed1c6d

brianjlai reviewed Aug 1, 2023

View reviewed changes

Merge branch 'master' into maxi297/p0-source-amplitude-cursor-datetim…

707127c

…e-formats

brianjlai approved these changes Aug 1, 2023

View reviewed changes

Automated Commit - Formatting Changes

425a7da

maxi297 added checklist-action-run and removed checklist-action-run labels Aug 1, 2023

girarda approved these changes Aug 1, 2023

View reviewed changes

lazebnyi approved these changes Aug 2, 2023

View reviewed changes

Merge branch 'master' into maxi297/p0-source-amplitude-cursor-datetim…

7e25160

…e-formats

bazarnov mentioned this pull request Aug 2, 2023

🐛 Source Amplitude: use latest CDK and fix average session length schema #28913

Closed

Merge branch 'master' into maxi297/p0-source-amplitude-cursor-datetim…

e072a2f

…e-formats

roman-yermilov-gl mentioned this pull request Sep 15, 2023

Source Amplitude: Stream_state saved in wrong format #29800

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Amplitude: fix datetime update and session length change #28942

🐛 Source Amplitude: fix datetime update and session length change #28942

maxi297 commented Aug 1, 2023 •

edited

Loading

github-actions bot commented Aug 1, 2023 •

edited by maxi297

Loading

maxi297 commented Aug 1, 2023 •

edited by github-actions bot

Loading

octavia-squidington-iii commented Aug 1, 2023

octavia-squidington-iii commented Aug 1, 2023

octavia-squidington-iii commented Aug 1, 2023

maxi297 Aug 1, 2023

octavia-squidington-iii commented Aug 1, 2023

octavia-squidington-iii commented Aug 1, 2023

octavia-squidington-iii commented Aug 1, 2023

brianjlai Aug 1, 2023

brianjlai Aug 1, 2023

brianjlai left a comment •

edited

Loading

octavia-squidington-iii commented Aug 1, 2023

girarda left a comment

octavia-squidington-iii commented Aug 2, 2023

octavia-squidington-iii commented Aug 2, 2023

octavia-squidington-iii commented Aug 2, 2023

🐛 Source Amplitude: fix datetime update and session length change #28942

Are you sure you want to change the base?

🐛 Source Amplitude: fix datetime update and session length change #28942

Conversation

maxi297 commented Aug 1, 2023 • edited Loading

What

How

github-actions bot commented Aug 1, 2023 • edited by maxi297 Loading

Before Merging a Connector Pull Request

maxi297 commented Aug 1, 2023 • edited by github-actions bot Loading

Build Failed

octavia-squidington-iii commented Aug 1, 2023

source-amplitude test report (commit 689657f37f) - ❌

octavia-squidington-iii commented Aug 1, 2023

source-amplitude test report (commit ff44713964) - ❌

octavia-squidington-iii commented Aug 1, 2023

source-amplitude test report (commit fb3e464fc0) - ❌

maxi297 Aug 1, 2023

Choose a reason for hiding this comment

octavia-squidington-iii commented Aug 1, 2023

source-amplitude test report (commit a92246ef04) - ❌

octavia-squidington-iii commented Aug 1, 2023

source-amplitude test report (commit b9e613b16b) - ❌

octavia-squidington-iii commented Aug 1, 2023

source-amplitude test report (commit eb9ff2b7e6) - ❌

brianjlai Aug 1, 2023

Choose a reason for hiding this comment

brianjlai Aug 1, 2023

Choose a reason for hiding this comment

brianjlai left a comment • edited Loading

Choose a reason for hiding this comment

octavia-squidington-iii commented Aug 1, 2023

source-amplitude test report (commit 9ed1c6defc) - ❌

girarda left a comment

Choose a reason for hiding this comment

octavia-squidington-iii commented Aug 2, 2023

source-amplitude test report (commit 425a7da08d) - ❌

octavia-squidington-iii commented Aug 2, 2023

source-amplitude test report (commit 7e25160417) - ❌

octavia-squidington-iii commented Aug 2, 2023

source-amplitude test report (commit e072a2fa1e) - ❌

maxi297 commented Aug 1, 2023 •

edited

Loading

github-actions bot commented Aug 1, 2023 •

edited by maxi297

Loading

maxi297 commented Aug 1, 2023 •

edited by github-actions bot

Loading

source-amplitude test report (commit `689657f37f`) - ❌

source-amplitude test report (commit `ff44713964`) - ❌

source-amplitude test report (commit `fb3e464fc0`) - ❌

source-amplitude test report (commit `a92246ef04`) - ❌

source-amplitude test report (commit `b9e613b16b`) - ❌

source-amplitude test report (commit `eb9ff2b7e6`) - ❌

brianjlai left a comment •

edited

Loading

source-amplitude test report (commit `9ed1c6defc`) - ❌

source-amplitude test report (commit `425a7da08d`) - ❌

source-amplitude test report (commit `7e25160417`) - ❌

source-amplitude test report (commit `e072a2fa1e`) - ❌