-
Notifications
You must be signed in to change notification settings - Fork 127
fallback to full table partitions for Timescale compressed hypertables #3318
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds fallback logic to full table partitions for Timescale compressed hypertables by introducing a new activity GetDefaultPartitionKeyForTables
that dynamically determines partition keys for tables during snapshot operations. For compressed hypertables in TimescaleDB, the system falls back to full table partitions instead of using ctid
-based partitioning.
- Removes hardcoded
supportsTidScans
boolean from various data structures and replaces it with dynamic partition key determination - Adds new protobuf messages and activity for determining default partition keys per table
- Implements TimescaleDB-specific logic to detect compressed hypertables and disable
ctid
scanning for them
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
File | Description |
---|---|
protos/flow.proto | Adds new proto messages for partition key determination and removes deprecated supports_tid_scans fields |
flow/workflows/snapshot_flow.go | Replaces static partition key logic with dynamic activity-based approach |
flow/connectors/postgres/qrep.go | Implements PostgreSQL-specific logic to detect TimescaleDB compressed hypertables |
flow/connectors/postgres/postgres.go | Removes supports_tid_scans field from snapshot export output |
flow/connectors/mysql/qrep.go | Adds stub implementation returning empty partition keys for MySQL |
flow/connectors/mongo/qrep.go | Adds stub implementation returning empty partition keys for MongoDB |
flow/connectors/core.go | Adds new interface method for partition key determination |
flow/activities/snapshot_activity.go | Implements the new activity and removes deprecated TID scan support fields |
Comments suppressed due to low confidence (1)
flow/connectors/mysql/qrep.go:25
- The removal of GetDataTypeOfWatermarkColumn function appears unrelated to the partition key changes described in the PR. This function removal should be in a separate commit or explained in the PR description.
func (c *MySqlConnector) GetQRepPartitions(
The PR adds a new activity,
GetDefaultPartitionKeyForTables,
during Snapshot, which is supposed to return the default "partition" key for each table if not overridden by the user. For Postgres, we check for PG14+ (when TID range scan was added), and if that is there, we also check if the Timescale extension is installed and for compressed hypertables. If any are found, disable the partition key for those.For Mongo/MySQL it currently returns an empty key for every table. But can be changed.