SAC-29835: Add sync logic for the tap #41

MuralidharT03 · 2025-12-24T05:23:51Z

Description of change

Implement sync workflow for the following streams of the tap
JIRA Link: https://qlik-dev.atlassian.net/browse/SAC-29835

Manual QA steps

Verify sync run
Verify sync for parent child streams
Run sync with state file

Risks

Rollback steps

revert this branch

AI generated code

https://internal.qlik.dev/general/ways-of-working/code-reviews/#guidelines-for-ai-generated-code

this PR has been written with the help of GitHub Copilot or another generative AI tool

Copilot

Pull request overview

This PR implements sync logic for the tap-trello connector by refactoring stream classes into a modular structure and adding support for parent-child stream relationships with proper bookmarking and pagination.

Refactored stream classes from a monolithic streams.py into individual modules under tap_trello/streams/
Added sync logic for boards, users, lists, actions, cards, and checklists streams with proper parent-child relationships
Implemented date windowing pagination for incremental streams and custom pagination for cards

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tap_trello/sync.py	Reorganized imports for better readability
tap_trello/streams/users.py	Created Users stream as child of Boards with FULL_TABLE replication
tap_trello/streams/lists.py	Created Lists stream as child of Boards with FULL_TABLE replication
tap_trello/streams/checklists.py	Created Checklists stream as child of Boards with FULL_TABLE replication
tap_trello/streams/cards.py	Created Cards stream with custom pagination and custom fields support
tap_trello/streams/boards.py	Created Boards stream as parent stream with FULL_TABLE replication
tap_trello/streams/actions.py	Created Actions stream with date windowing and INCREMENTAL replication
tap_trello/streams/abstracts.py	Moved shared stream classes and mixins from streams.py, added backward-compatible aliases
tap_trello/streams/init.py	Updated to import and expose new stream classes
tap_trello/streams.py	Removed entire file as content moved to abstracts.py and individual stream files
tap_trello/schema.py	Added logic to handle parent references as both strings and class objects
tap_trello/client.py	Added TrelloRateLimitError to backoff exception handling
setup.py	Updated requests-oauthlib dependency from 1.3.0 to 2.0.0
.circleci/config.yml	Added unset command for USE_STITCH_BACKEND in test job

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-24T05:24:53Z

tap_trello/streams/abstracts.py

+        # adjusting window to lookback 1 day
+        adjusted_window_start = utils.strftime(utils.strptime_to_utc(window_start)-timedelta(days=1))
+
+        start_date = self.config.get('start_date')
+        end_date = self.config.get('end_date', window_end)
+
+        window_start = utils.strptime_to_utc(max(adjusted_window_start, start_date))
+        sub_window_end = sub_window_end and  utils.strptime_to_utc(min(sub_window_end, end_date))
+        window_end = utils.strptime_to_utc(min(window_end, end_date))
+
+        return window_start, sub_window_end, window_end
+


The function does not handle the case where window_start, sub_window_end, or window_end bookmarks are None before using them in datetime operations on lines 99-106. If any of these are None, the subsequent utils.strptime_to_utc() calls will fail.

Suggested change

# adjusting window to lookback 1 day

adjusted_window_start = utils.strftime(utils.strptime_to_utc(window_start)-timedelta(days=1))

start_date = self.config.get('start_date')

end_date = self.config.get('end_date', window_end)

window_start = utils.strptime_to_utc(max(adjusted_window_start, start_date))

sub_window_end = sub_window_end and utils.strptime_to_utc(min(sub_window_end, end_date))

window_end = utils.strptime_to_utc(min(window_end, end_date))

return window_start, sub_window_end, window_end

start_date = self.config.get('start_date')

config_end_date = self.config.get('end_date')

# Determine base string values to use for the window bounds.

# Prefer bookmarks when present, otherwise fall back to config values.

base_start_str = window_start or start_date

base_end_str = window_end or config_end_date or start_date

# If we don't have enough information to build a window, return empty state.

if base_start_str is None or base_end_str is None:

return None, None, None

# Adjust the window to look back 1 day from the base start.

adjusted_window_start_str = utils.strftime(

utils.strptime_to_utc(base_start_str) - timedelta(days=1)

)

# The effective end_date string is either the configured end_date

# or the resolved base_end_str when no explicit config is provided.

end_date_str = config_end_date or base_end_str

# Convert to datetimes, making sure to only compare non-None strings.

if start_date is not None:

window_start_dt_str = max(adjusted_window_start_str, start_date)

else:

window_start_dt_str = adjusted_window_start_str

window_start = utils.strptime_to_utc(window_start_dt_str)

sub_window_end_dt = None

if sub_window_end is not None:

if end_date_str is not None:

sub_window_end_dt_str = min(sub_window_end, end_date_str)

else:

sub_window_end_dt_str = sub_window_end

sub_window_end_dt = utils.strptime_to_utc(sub_window_end_dt_str)

if end_date_str is not None:

window_end_dt_str = min(base_end_str, end_date_str)

else:

window_end_dt_str = base_end_str

window_end = utils.strptime_to_utc(window_end_dt_str)

return window_start, sub_window_end_dt, window_end

tap_trello/streams/abstracts.py

tap_trello/streams/cards.py

…logic

vishalp-dev · 2026-01-02T10:02:17Z

tap_trello/streams/cards.py

+        cards_response_size = self.config.get('cards_response_size')
+        self.MAX_API_RESPONSE_SIZE = int(cards_response_size) if cards_response_size else self.MAX_API_RESPONSE_SIZE


This could be simplified as

cards_response_size = int(self.config.get('cards_response_size') or self.MAX_API_RESPONSE_SIZE)

If this cannot be more than 1000, should this be validated and set to 1000 if the input value provided is more than 1000?

vishalp-dev · 2026-01-02T10:05:37Z

tap_trello/streams/cards.py

+            if self.MAX_API_RESPONSE_SIZE and len(records) > self.MAX_API_RESPONSE_SIZE:
+                raise Exception(
+                    ("{}: Number of records returned is greater than the requested API response size of {}.").format(
+                        self.stream_id,
+                        self.MAX_API_RESPONSE_SIZE)
+                )


can you please explain why this is a non tolerable exception?

vishalp-dev · 2026-01-02T12:03:16Z

tap_trello/streams/cards.py

+        while True:
+


use a flag variable instead of while True

vishalp-dev

Please Review the suggestions

Add sync logic for the tap

deb9183

MuralidharT03 requested a review from Copilot December 24, 2025 05:23

Copilot AI reviewed Dec 24, 2025

View reviewed changes

MuralidharT03 added 2 commits December 29, 2025 14:33

Updated sync logic

77301a2

Fixed naming convention

936b557

MuralidharT03 requested a review from vishalp-dev December 29, 2025 11:58

MuralidharT03 added 2 commits December 29, 2025 19:46

Fixed pylint errors and refactored duplicate code

7c298a0

Merge branch 'SAC-29830-Create-Schema' into SAC-29835-implement-sync-…

e211a86

…logic

vishalp-dev reviewed Jan 2, 2026

View reviewed changes

tap_trello/streams/cards.py

Comment on lines +30 to +31

while True:

Copy link

Member

vishalp-dev Jan 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use a flag variable instead of while True

vishalp-dev reviewed Jan 2, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

SAC-29835: Add sync logic for the tap #41

SAC-29835: Add sync logic for the tap #41

Uh oh!

MuralidharT03 commented Dec 24, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Dec 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vishalp-dev Jan 2, 2026 •

edited

Loading

Uh oh!

vishalp-dev Jan 2, 2026

Uh oh!

vishalp-dev Jan 2, 2026

Uh oh!

vishalp-dev left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-        # adjusting window to lookback 1 day
-        adjusted_window_start = utils.strftime(utils.strptime_to_utc(window_start)-timedelta(days=1))
-        start_date = self.config.get('start_date')
-        end_date = self.config.get('end_date', window_end)
-        window_start = utils.strptime_to_utc(max(adjusted_window_start, start_date))
-        sub_window_end = sub_window_end and  utils.strptime_to_utc(min(sub_window_end, end_date))
-        window_end = utils.strptime_to_utc(min(window_end, end_date))
-        return window_start, sub_window_end, window_end
+        start_date = self.config.get('start_date')
+        config_end_date = self.config.get('end_date')
+        # Determine base string values to use for the window bounds.
+        # Prefer bookmarks when present, otherwise fall back to config values.
+        base_start_str = window_start or start_date
+        base_end_str = window_end or config_end_date or start_date
+        # If we don't have enough information to build a window, return empty state.
+        if base_start_str is None or base_end_str is None:
+            return None, None, None
+        # Adjust the window to look back 1 day from the base start.
+        adjusted_window_start_str = utils.strftime(
+            utils.strptime_to_utc(base_start_str) - timedelta(days=1)
+        )
+        # The effective end_date string is either the configured end_date
+        # or the resolved base_end_str when no explicit config is provided.
+        end_date_str = config_end_date or base_end_str
+        # Convert to datetimes, making sure to only compare non-None strings.
+        if start_date is not None:
+            window_start_dt_str = max(adjusted_window_start_str, start_date)
+        else:
+            window_start_dt_str = adjusted_window_start_str
+        window_start = utils.strptime_to_utc(window_start_dt_str)
+        sub_window_end_dt = None
+        if sub_window_end is not None:
+            if end_date_str is not None:
+                sub_window_end_dt_str = min(sub_window_end, end_date_str)
+            else:
+                sub_window_end_dt_str = sub_window_end
+            sub_window_end_dt = utils.strptime_to_utc(sub_window_end_dt_str)
+        if end_date_str is not None:
+            window_end_dt_str = min(base_end_str, end_date_str)
+        else:
+            window_end_dt_str = base_end_str
+        window_end = utils.strptime_to_utc(window_end_dt_str)
+        return window_start, sub_window_end_dt, window_end

		cards_response_size = self.config.get('cards_response_size')
		self.MAX_API_RESPONSE_SIZE = int(cards_response_size) if cards_response_size else self.MAX_API_RESPONSE_SIZE

SAC-29835: Add sync logic for the tap #41

Are you sure you want to change the base?

SAC-29835: Add sync logic for the tap #41

Uh oh!

Conversation

MuralidharT03 commented Dec 24, 2025

Description of change

Manual QA steps

Risks

Rollback steps

AI generated code

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vishalp-dev Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vishalp-dev Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

vishalp-dev Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

vishalp-dev left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vishalp-dev Jan 2, 2026 •

edited

Loading