Skip to content

fix(taps): ignore start date for default non-timestamp keys - #3798

Open
akashmalbari wants to merge 1 commit into
meltano:mainfrom
akashmalbari:fix/1677-ignore-start-date-non-timestamp-keys
Open

akashmalbari wants to merge 1 commit into
meltano:mainfrom
akashmalbari:fix/1677-ignore-start-date-non-timestamp-keys

Conversation

@akashmalbari

@akashmalbari akashmalbari commented Sep 25, 2026 •

Copy link
Copy Markdown

Summary

  • prevent the default start_date from seeding an incremental bookmark when the replication key is not timestamp-typed
  • retain support for streams that explicitly override compare_start_date to convert a non-timestamp key
  • add a regression case for integer replication keys without prior state

Fixes #1677

Validation

  • uv run pytest tests/core/test_streams.py tests/core/test_stream_state_manager.py (89 passed)
  • uvx prek run --files singer_sdk/streams/core.py tests/core/test_streams.py (passed)
  • uv run mypy singer_sdk/streams/core.py (passed)
  • uv run ty check singer_sdk/streams/core.py (passed)

Compatibility and privacy/security impact

This corrects the default behavior for non-timestamp replication keys. Connectors that intentionally derive a non-timestamp bookmark from start_date can preserve that behavior by overriding compare_start_date. No credentials, tokens, records, or logging behavior are changed.

Summary by Sourcery

Ignore the default start date when initializing non-timestamp replication keys unless a stream explicitly supports converting it.

Bug Fixes:

  • Prevent the default start date from initializing incremental bookmarks for non-timestamp replication keys.
  • Preserve start-date conversion for streams that explicitly override the comparison behavior.

Tests:

  • Add regression coverage for integer replication keys without existing state.

@sourcery-ai

sourcery-ai Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor
Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

The PR fixes incremental bookmark initialization by ignoring the default timestamp start_date for non-timestamp replication keys, while retaining compatibility for streams that explicitly override compare_start_date, and adds regression coverage for the no-prior-state case.

Sequence diagram for replication bookmark initialization

sequenceDiagram
    participant Stream
    participant StateManager

    Stream->>Stream: _write_starting_replication_value(context)
    alt non-timestamp key and default compare_start_date
        Stream->>Stream: config.pop(start_date, None)
    else timestamp key or overridden compare_start_date
        Stream->>Stream: retain start_date
    end
    Stream->>StateManager: write_starting_replication_value(context, replication_method, replication_key, config, compare_start_date)
Loading

File-Level Changes

Change Details Files
Avoid seeding non-timestamp replication keys with the default ISO start_date while preserving explicit conversion behavior.
  • Copy stream configuration before initializing the starting replication value.
  • Remove start_date when the key is non-timestamp-typed and the stream uses the base compare_start_date implementation.
  • Continue passing the configured comparison callback so streams overriding compare_start_date retain custom conversion support.
singer_sdk/streams/core.py
Update regression coverage for integer-like replication keys without prior state.
  • Assert that a non-timestamp key does not receive the default start date when applying the catalog.
tests/core/test_streams.py

Assessment against linked issues

Issue Objective Addressed Explanation
#1677 Prevent the configured start_date from seeding an incremental bookmark when the replication key is not timestamp-typed, avoiding database conversion errors for integer or other non-datetime keys. ✅
#1677 Preserve support for streams that intentionally convert start_date into a non-timestamp replication value by overriding compare_start_date. ✅
#1677 Add regression coverage for an initial incremental run using a non-timestamp replication key without prior state. ✅

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've reviewed your changes and they look great!

Sourcery assessment

Needs a human reviewer. This changes the starting replication value written to state for non-timestamp keys, so an incorrect decision could cause records to be skipped or an unnecessarily broad sync to run. Reverting would stop the behavior, but state already written may need to be reset or the affected sync rerun to repair it.


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

@codecov

codecov Bot commented Sep 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.44%. Comparing base (9e68e0a) to head (027a05a).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3798      +/-   ##
==========================================
- Coverage   94.47%   94.44%   -0.03%     
==========================================
  Files          74       74              
  Lines        6295     6298       +3     
  Branches      770      771       +1     
==========================================
+ Hits         5947     5948       +1     
- Misses        260      261       +1     
- Partials       88       89       +1     
Flag Coverage Δ
core 83.13% <100.00%> (-0.03%) ⬇️
end-to-end 76.24% <100.00%> (+0.01%) ⬆️
optional-components 44.98% <0.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed

codspeed Bot commented Sep 25, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 14 untouched benchmarks


Comparing akashmalbari:fix/1677-ignore-start-date-non-timestamp-keys (027a05a) with main (9e68e0a)

Open in CodSpeed

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: start_date from meltano.yml used even if replication-key column is not date-time

1 participant