Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🐛 Source Amplitude: fix datetime update and session length change #28942

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
Open

🐛 Source Amplitude: fix datetime update and session length change #28942

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
9c3e2c1
[ISSUE #28782] support multiple cursor field datetime formats
maxi297 Aug 1, 2023
689657f
Fix amplitude datetime update and session length change
maxi297 Aug 1, 2023
ff44713
bump version and changelogs
maxi297 Aug 1, 2023
fb3e464
Update schema
maxi297 Aug 1, 2023
953bc45
Merge branch 'master' into maxi297/p0-source-amplitude-cursor-datetim…
maxi297 Aug 1, 2023
1fe733e
Automated Commit - Formatting Changes
maxi297 Aug 1, 2023
a92246e
Bump minor as this is a breaking change now
maxi297 Aug 1, 2023
b9e613b
Retrigger pipeline
maxi297 Aug 1, 2023
eb9ff2b
Force CDK version
maxi297 Aug 1, 2023
9ed1c6d
Retrigger pipeline
maxi297 Aug 1, 2023
707127c
Merge branch 'master' into maxi297/p0-source-amplitude-cursor-datetim…
maxi297 Aug 1, 2023
425a7da
Automated Commit - Formatting Changes
maxi297 Aug 1, 2023
7e25160
Merge branch 'master' into maxi297/p0-source-amplitude-cursor-datetim…
maxi297 Aug 2, 2023
e072a2f
Merge branch 'master' into maxi297/p0-source-amplitude-cursor-datetim…
maxi297 Aug 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-cdk/python/unit_tests/sources/declarative/incremental/test_datetime_based_cursor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -618,7 +618,7 @@ def test_cursor_granularity_but_no_step():
)


def test_given_multiple_cursor_datetime_format_then_slice_using_first_format():
def test_given_multiple_cursor_datetime_format_then_slice_using_datetime_format():
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just cleaning up naming but doesn't affect this PR

cursor = DatetimeBasedCursor(
start_datetime=MinMaxDatetime("2021-01-01", parameters={}),
end_datetime=MinMaxDatetime("2023-01-10", parameters={}),
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-amplitude/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,5 @@ COPY source_amplitude ./source_amplitude
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.2.4
LABEL io.airbyte.version=0.2.5
LABEL io.airbyte.name=airbyte/source-amplitude
46 changes: 23 additions & 23 deletions airbyte-integrations/connectors/source-amplitude/integration_tests/expected_records.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,29 +13,29 @@
{"stream":"active_users","data":{"date":"2023-02-13","statistics":{"(none)":1,"Ukraine":0}},"emitted_at":1676633494312}
{"stream":"active_users","data":{"date":"2023-02-14","statistics":{"(none)":1,"Ukraine":0}},"emitted_at":1676633494312}
{"stream":"active_users","data":{"date":"2023-02-15","statistics":{"(none)":1,"Ukraine":0}},"emitted_at":1676633494312}
{"stream":"average_session_length","data":{"date":"2023-02-01","length":0},"emitted_at":1677160664431}
{"stream":"average_session_length","data":{"date":"2023-02-02","length":0},"emitted_at":1677160664432}
{"stream":"average_session_length","data":{"date":"2023-02-03","length":0},"emitted_at":1677160664434}
{"stream":"average_session_length","data":{"date":"2023-02-04","length":0},"emitted_at":1677160664435}
{"stream":"average_session_length","data":{"date":"2023-02-05","length":0},"emitted_at":1677160664436}
{"stream":"average_session_length","data":{"date":"2023-02-06","length":0},"emitted_at":1677160664438}
{"stream":"average_session_length","data":{"date":"2023-02-07","length":0},"emitted_at":1677160664439}
{"stream":"average_session_length","data":{"date":"2023-02-08","length":0},"emitted_at":1677160664441}
{"stream":"average_session_length","data":{"date":"2023-02-09","length":0},"emitted_at":1677160664442}
{"stream":"average_session_length","data":{"date":"2023-02-10","length":0},"emitted_at":1677160664443}
{"stream":"average_session_length","data":{"date":"2023-02-11","length":0},"emitted_at":1677160664445}
{"stream":"average_session_length","data":{"date":"2023-02-12","length":0},"emitted_at":1677160664446}
{"stream":"average_session_length","data":{"date":"2023-02-13","length":0},"emitted_at":1677160664447}
{"stream":"average_session_length","data":{"date":"2023-02-14","length":0},"emitted_at":1677160664449}
{"stream":"average_session_length","data":{"date":"2023-02-15","length":0},"emitted_at":1677160664450}
{"stream":"average_session_length","data":{"date":"2023-02-16","length":0},"emitted_at":1677160664803}
{"stream":"average_session_length","data":{"date":"2023-02-17","length":0},"emitted_at":1677160664805}
{"stream":"average_session_length","data":{"date":"2023-02-18","length":0},"emitted_at":1677160664806}
{"stream":"average_session_length","data":{"date":"2023-02-19","length":0},"emitted_at":1677160664812}
{"stream":"average_session_length","data":{"date":"2023-02-20","length":0},"emitted_at":1677160664814}
{"stream":"average_session_length","data":{"date":"2023-02-21","length":0},"emitted_at":1677160664815}
{"stream":"average_session_length","data":{"date":"2023-02-22","length":0},"emitted_at":1677160664816}
{"stream":"average_session_length","data":{"date":"2023-02-23","length":0},"emitted_at":1677160664818}
{"stream":"average_session_length","data":{"date":"2023-02-01T00:00:00","length":0},"emitted_at":1677160664431}
{"stream":"average_session_length","data":{"date":"2023-02-02T00:00:00","length":0},"emitted_at":1677160664432}
{"stream":"average_session_length","data":{"date":"2023-02-03T00:00:00","length":0},"emitted_at":1677160664434}
{"stream":"average_session_length","data":{"date":"2023-02-04T00:00:00","length":0},"emitted_at":1677160664435}
{"stream":"average_session_length","data":{"date":"2023-02-05T00:00:00","length":0},"emitted_at":1677160664436}
{"stream":"average_session_length","data":{"date":"2023-02-06T00:00:00","length":0},"emitted_at":1677160664438}
{"stream":"average_session_length","data":{"date":"2023-02-07T00:00:00","length":0},"emitted_at":1677160664439}
{"stream":"average_session_length","data":{"date":"2023-02-08T00:00:00","length":0},"emitted_at":1677160664441}
{"stream":"average_session_length","data":{"date":"2023-02-09T00:00:00","length":0},"emitted_at":1677160664442}
{"stream":"average_session_length","data":{"date":"2023-02-10T00:00:00","length":0},"emitted_at":1677160664443}
{"stream":"average_session_length","data":{"date":"2023-02-11T00:00:00","length":0},"emitted_at":1677160664445}
{"stream":"average_session_length","data":{"date":"2023-02-12T00:00:00","length":0},"emitted_at":1677160664446}
{"stream":"average_session_length","data":{"date":"2023-02-13T00:00:00","length":0},"emitted_at":1677160664447}
{"stream":"average_session_length","data":{"date":"2023-02-14T00:00:00","length":0},"emitted_at":1677160664449}
{"stream":"average_session_length","data":{"date":"2023-02-15T00:00:00","length":0},"emitted_at":1677160664450}
{"stream":"average_session_length","data":{"date":"2023-02-16T00:00:00","length":0},"emitted_at":1677160664803}
{"stream":"average_session_length","data":{"date":"2023-02-17T00:00:00","length":0},"emitted_at":1677160664805}
{"stream":"average_session_length","data":{"date":"2023-02-18T00:00:00","length":0},"emitted_at":1677160664806}
{"stream":"average_session_length","data":{"date":"2023-02-19T00:00:00","length":0},"emitted_at":1677160664812}
{"stream":"average_session_length","data":{"date":"2023-02-20T00:00:00","length":0},"emitted_at":1677160664814}
{"stream":"average_session_length","data":{"date":"2023-02-21T00:00:00","length":0},"emitted_at":1677160664815}
{"stream":"average_session_length","data":{"date":"2023-02-22T00:00:00","length":0},"emitted_at":1677160664816}
{"stream":"average_session_length","data":{"date":"2023-02-23T00:00:00","length":0},"emitted_at":1677160664818}
{"stream":"events","data":{"$insert_id":"google-ad-4651612872-643022056303-DESKTOP-2023-01-31","$insert_key":null,"$schema":null,"adid":null,"amplitude_attribution_ids":null,"amplitude_event_type":null,"amplitude_id":550106004607,"app":434735,"city":null,"client_event_time":"2023-01-31T08:00:00+00:00","client_upload_time":"2023-02-01T12:02:17.685000+00:00","country":null,"data":{"user_properties_updated":true,"vacuum_source_id":"5955","group_first_event":{},"group_ids":{}},"data_type":"event","device_brand":null,"device_carrier":null,"device_family":null,"device_id":"google-ad-4651612872-643022056303","device_manufacturer":null,"device_model":null,"device_type":null,"dma":null,"event_id":82672889,"event_properties":{"ad_metrics.cost":0.055104,"campaign_advertising_channel_type":"DISPLAY","ad_segment_device":"DESKTOP","ad_metrics.impressions":202,"ad_group_type":"DISPLAY_STANDARD","campaign_name":"Brand awareness and reach-Display-1","ad_group_name":"Ad group 1","ad_id":643022056303,"campaign_start_date":"2022-12-28","final_url":"https://airbyte.com","ad_platform":"google","campaign_end_date":"2037-12-30","ad_metrics.clicks":0,"ad_group_id":144799120517,"ad_metrics.conversions":0,"ad_metrics.interactions":0,"campaign_id":19410069806},"event_time":"2023-01-31T08:00:00+00:00","event_type":"Daily Ad Metrics","global_user_properties":null,"group_properties":{},"groups":{},"idfa":null,"ip_address":null,"is_attribution_event":null,"language":null,"library":"google_ads","location_lat":null,"location_lng":null,"os_name":null,"os_version":null,"partner_id":null,"paying":null,"plan":{},"platform":null,"processed_time":"2023-02-01T12:02:40.918000+00:00","region":null,"sample_rate":null,"server_received_time":"2023-02-01T12:02:17.685000+00:00","server_upload_time":"2023-02-01T12:02:36.650000+00:00","session_id":-1,"source_id":null,"start_version":null,"user_creation_time":null,"user_id":null,"user_properties":{},"uuid":"53e385c4-a228-11ed-8e51-171d53d06523","version_name":null},"emitted_at":1677160703462}
{"stream":"events","data":{"$insert_id":"google-ad-4651612872-643022056303-TABLET-2023-01-31","$insert_key":null,"$schema":null,"adid":null,"amplitude_attribution_ids":null,"amplitude_event_type":null,"amplitude_id":550106004607,"app":434735,"city":null,"client_event_time":"2023-01-31T08:00:00+00:00","client_upload_time":"2023-02-01T12:02:17.685000+00:00","country":null,"data":{"vacuum_source_id":"5955","group_first_event":{},"group_ids":{}},"data_type":"event","device_brand":null,"device_carrier":null,"device_family":null,"device_id":"google-ad-4651612872-643022056303","device_manufacturer":null,"device_model":null,"device_type":null,"dma":null,"event_id":273196606,"event_properties":{"ad_metrics.cost":0.10283,"campaign_advertising_channel_type":"DISPLAY","ad_segment_device":"TABLET","ad_metrics.impressions":188,"ad_group_type":"DISPLAY_STANDARD","campaign_name":"Brand awareness and reach-Display-1","ad_group_name":"Ad group 1","ad_id":643022056303,"campaign_start_date":"2022-12-28","final_url":"https://airbyte.com","ad_platform":"google","campaign_end_date":"2037-12-30","ad_metrics.clicks":0,"ad_group_id":144799120517,"ad_metrics.conversions":0,"ad_metrics.interactions":0,"campaign_id":19410069806},"event_time":"2023-01-31T08:00:00+00:00","event_type":"Daily Ad Metrics","global_user_properties":null,"group_properties":{},"groups":{},"idfa":null,"ip_address":null,"is_attribution_event":null,"language":null,"library":"google_ads","location_lat":null,"location_lng":null,"os_name":null,"os_version":null,"partner_id":null,"paying":null,"plan":{},"platform":null,"processed_time":"2023-02-01T12:02:40.918000+00:00","region":null,"sample_rate":null,"server_received_time":"2023-02-01T12:02:17.685000+00:00","server_upload_time":"2023-02-01T12:02:36.650000+00:00","session_id":-1,"source_id":null,"start_version":null,"user_creation_time":null,"user_id":null,"user_properties":{},"uuid":"53e39277-a228-11ed-9400-171d53d06523","version_name":null},"emitted_at":1677160703465}
{"stream":"events","data":{"$insert_id":"google-ad-4651612872-643022056303-MOBILE-2023-01-31","$insert_key":null,"$schema":null,"adid":null,"amplitude_attribution_ids":null,"amplitude_event_type":null,"amplitude_id":550106004607,"app":434735,"city":null,"client_event_time":"2023-01-31T08:00:00+00:00","client_upload_time":"2023-02-01T12:02:17.685000+00:00","country":null,"data":{"vacuum_source_id":"5955","group_first_event":{},"group_ids":{}},"data_type":"event","device_brand":null,"device_carrier":null,"device_family":null,"device_id":"google-ad-4651612872-643022056303","device_manufacturer":null,"device_model":null,"device_type":null,"dma":null,"event_id":695784978,"event_properties":{"ad_metrics.cost":1.781222,"campaign_advertising_channel_type":"DISPLAY","ad_segment_device":"MOBILE","ad_metrics.impressions":2406,"ad_group_type":"DISPLAY_STANDARD","campaign_name":"Brand awareness and reach-Display-1","ad_group_name":"Ad group 1","ad_id":643022056303,"campaign_start_date":"2022-12-28","final_url":"https://airbyte.com","ad_platform":"google","campaign_end_date":"2037-12-30","ad_metrics.clicks":3,"ad_group_id":144799120517,"ad_metrics.conversions":0,"ad_metrics.interactions":3,"campaign_id":19410069806},"event_time":"2023-01-31T08:00:00+00:00","event_type":"Daily Ad Metrics","global_user_properties":null,"group_properties":{},"groups":{},"idfa":null,"ip_address":null,"is_attribution_event":null,"language":null,"library":"google_ads","location_lat":null,"location_lng":null,"os_name":null,"os_version":null,"partner_id":null,"paying":null,"plan":{},"platform":null,"processed_time":"2023-02-01T12:02:40.918000+00:00","region":null,"sample_rate":null,"server_received_time":"2023-02-01T12:02:17.685000+00:00","server_upload_time":"2023-02-01T12:02:36.650000+00:00","session_id":-1,"source_id":null,"start_version":null,"user_creation_time":null,"user_id":null,"user_properties":{},"uuid":"53e38ca8-a228-11ed-a707-171d53d06523","version_name":null},"emitted_at":1677160703491}
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-amplitude/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: fa9f58c6-2d03-4237-aaa4-07d75e0c1396
dockerImageTag: 0.2.4
dockerImageTag: 0.2.5
dockerRepository: airbyte/source-amplitude
githubIssueLabel: source-amplitude
icon: amplitude.svg
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-amplitude/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from setuptools import find_packages, setup

MAIN_REQUIREMENTS = [
"airbyte-cdk==0.33.0",
"airbyte-cdk",
]

TEST_REQUIREMENTS = [
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-amplitude/source_amplitude/components.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ def extract_records(self, response: requests.Response) -> List[Record]:
series = response_data.get("series", [])
if len(series) > 0:
series = series[0] # get the nested list
return [{"date": date, "length": length} for date, length in zip(response_data["xValues"], series)]
return [{"date": date, "length": length["value"]} for date, length in zip(response_data["xValues"], series)]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why doesn't the change from using length to the length's value key result in a change to expected records? I'm a little confused where this value key comes from based on the API docs: https://www.docs.developers.amplitude.com/analytics/apis/dashboard-rest-api/#response_3

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Synced with Maxime, Amplitude response has also changed (but not documented) that has changed what was original an array of numbers:

[12.0, 15.0]

to an array of objects:

[{ "setId": "", "value": 12.0 }, { "setId": "", "value": 15.0 }]

We need this to retain compatibility with our existing connector behavior so this is all good.

return []


Expand Down
5 changes: 5 additions & 0 deletions airbyte-integrations/connectors/source-amplitude/source_amplitude/manifest.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,11 @@ definitions:
extractor:
type: CustomRecordExtractor
class_name: source_amplitude.components.AverageSessionLengthRecordExtractor
incremental_sync:
$ref: "#/definitions/datetime_incremental_sync"
cursor_datetime_formats:
- "%Y-%m-%dT%H:%M:%S"
- "%Y-%m-%d"
$parameters:
name: "average_session_length"
primary_key: "date"
Expand Down
2 changes: 1 addition & 1 deletion ...grations/connectors/source-amplitude/source_amplitude/schemas/average_session_length.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"properties": {
"date": {
"type": ["null", "string"],
"format": "date"
"format": "date-time"
},
"length": {
"type": ["null", "number"]
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-amplitude/unit_tests/test_custom_extractors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
AverageSessionLengthRecordExtractor,
{
"xValues": ["2019-05-23", "2019-05-24"],
"series": [[2, 6]],
"series": [[{"setId": "", "value": 2.0}, {"setId": "", "value": 6.0}]],
"seriesCollapsed": [[0]],
"seriesLabels": [0],
"seriesMeta": [{"segmentIndex": 0}],
Expand Down
Loading
Loading