Skip to content

fallback to full table partitions for Timescale compressed hypertables #3318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 7, 2025

Conversation

heavycrystal
Copy link
Contributor

The PR adds a new activity, GetDefaultPartitionKeyForTables, during Snapshot, which is supposed to return the default "partition" key for each table if not overridden by the user. For Postgres, we check for PG14+ (when TID range scan was added), and if that is there, we also check if the Timescale extension is installed and for compressed hypertables. If any are found, disable the partition key for those.

For Mongo/MySQL it currently returns an empty key for every table. But can be changed.

Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds fallback logic to full table partitions for Timescale compressed hypertables by introducing a new activity GetDefaultPartitionKeyForTables that dynamically determines partition keys for tables during snapshot operations. For compressed hypertables in TimescaleDB, the system falls back to full table partitions instead of using ctid-based partitioning.

  • Removes hardcoded supportsTidScans boolean from various data structures and replaces it with dynamic partition key determination
  • Adds new protobuf messages and activity for determining default partition keys per table
  • Implements TimescaleDB-specific logic to detect compressed hypertables and disable ctid scanning for them

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
protos/flow.proto Adds new proto messages for partition key determination and removes deprecated supports_tid_scans fields
flow/workflows/snapshot_flow.go Replaces static partition key logic with dynamic activity-based approach
flow/connectors/postgres/qrep.go Implements PostgreSQL-specific logic to detect TimescaleDB compressed hypertables
flow/connectors/postgres/postgres.go Removes supports_tid_scans field from snapshot export output
flow/connectors/mysql/qrep.go Adds stub implementation returning empty partition keys for MySQL
flow/connectors/mongo/qrep.go Adds stub implementation returning empty partition keys for MongoDB
flow/connectors/core.go Adds new interface method for partition key determination
flow/activities/snapshot_activity.go Implements the new activity and removes deprecated TID scan support fields
Comments suppressed due to low confidence (1)

flow/connectors/mysql/qrep.go:25

  • The removal of GetDataTypeOfWatermarkColumn function appears unrelated to the partition key changes described in the PR. This function removal should be in a separate commit or explained in the PR description.
func (c *MySqlConnector) GetQRepPartitions(

@heavycrystal heavycrystal merged commit 3e957f2 into main Aug 7, 2025
20 of 23 checks passed
@heavycrystal heavycrystal deleted the feat/timescale-compression-ctid-fallback branch August 7, 2025 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants