Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement pagination for Forms stream #260

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Implement pagination for Forms stream #260

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
87820dc
Implement pagination for forms stream
Jul 25, 2024
619a93e
Update integration tests
Jul 25, 2024
1611707
Minor fixes
Jul 25, 2024
024e613
Bump version 3.1.0
Jul 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 22 additions & 9 deletions tap_hubspot/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -995,20 +995,33 @@ def sync_forms(STATE, ctx):

LOGGER.info("sync_forms from %s", start)

data = request(get_url("forms")).json()
time_extracted = utils.now()

with Transformer(UNIX_MILLISECONDS_INTEGER_DATETIME_PARSING) as bumble_bee:
# To handle records updated between start of the table sync and the end,
# store the current sync start in the state and not move the bookmark past this value.
sync_start_time = utils.now()
for row in data:
record = bumble_bee.transform(lift_properties_and_versions(row), schema, mdata)

if record[bookmark_key] >= start:
singer.write_record("forms", record, catalog.get('stream_alias'), time_extracted=time_extracted)
if record[bookmark_key] >= max_bk_value:
max_bk_value = record[bookmark_key]
# Check for pagination
params = {'formTypes': 'ALL',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any significance of adding formTypes=ALL in params?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is optional parameter, we can remove it.

'limit': 100,
'offset': 0}
while True:
LOGGER.info("Limit: %d, Offset: %d", params['limit'], params['offset'])
# Get next page URL
data = request(get_url("forms"), params).json()
time_extracted = utils.now()

for row in data:
record = bumble_bee.transform(lift_properties_and_versions(row), schema, mdata)

if record[bookmark_key] >= start:
singer.write_record("forms", record, catalog.get('stream_alias'), time_extracted=time_extracted)
if record[bookmark_key] >= max_bk_value:
max_bk_value = record[bookmark_key]

if len(data) == 100:
params['offset'] += 100
else:
break

# Don't bookmark past the start of this sync to account for updated records during the sync.
new_bookmark = min(utils.strptime_to_utc(max_bk_value), sync_start_time)
Expand Down
1 change: 1 addition & 0 deletions tests/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ def expected_metadata(self): # DOCS_BUG https://stitchdata.atlassian.net/browse
self.PRIMARY_KEYS: {"guid"},
self.REPLICATION_METHOD: self.INCREMENTAL,
self.REPLICATION_KEYS: {"updatedAt"},
self.EXPECTED_PAGE_SIZE: 100,
self.OBEYS_START_DATE: True
},
"owners": {
Expand Down
1 change: 1 addition & 0 deletions tests/base_hubspot.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ def expected_metadata(cls): # DOCS_BUG https://stitchdata.atlassian.net/browse/
BaseCase.PRIMARY_KEYS: {"guid"},
BaseCase.REPLICATION_METHOD: BaseCase.INCREMENTAL,
BaseCase.REPLICATION_KEYS: {"updatedAt"},
BaseCase.API_LIMIT: 100,
BaseCase.OBEYS_START_DATE: True
},
"owners": {
Expand Down