Skip to content

feat: add table reconciliation utilities #6519

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 17, 2025

Conversation

WenyXu
Copy link
Member

@WenyXu WenyXu commented Jul 14, 2025

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

#6429

What's changed and what's your intention?

This PR introduces utilities for table reconciliation and column metadata validation.

The new utilities provide three main functions:

  1. check_column_metadatas_consistent() - Verifies consistency across regions.
  2. resolve_column_metadatas_with_use_metasrv_strategy() - Resolves inconsistencies in column metadata, uses metasrv as source of truth.
  3. resolve_column_metadatas_with_use_latest_strategy() - Resolves inconsistencies in column metadata, uses highest schema version.
  4. check_column_metadata_invariants - Checks schema invariants

Ensures schema invariants are maintained:

  • Primary key columns must remain exactly the same in name and column id.
  • Timestamp column must remain exactly the same in name and column id.
  • All partition key columns must be present in new column metadata

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.
  • API changes are backward compatible.
  • Schema or data changes are backward compatible.

@WenyXu WenyXu requested a review from fengjiachun July 14, 2025 08:04
@github-actions github-actions bot added the docs-not-required This change does not impact docs. label Jul 14, 2025
Signed-off-by: WenyXu <wenymedia@gmail.com>
@WenyXu WenyXu force-pushed the feat/table-reconciliation-utils branch from 0678502 to 038bb99 Compare July 14, 2025 08:17
@WenyXu WenyXu marked this pull request as ready for review July 14, 2025 08:20
@WenyXu WenyXu requested review from MichaelScofield and a team as code owners July 14, 2025 08:20
WenyXu added 2 commits July 14, 2025 08:42
Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
@WenyXu WenyXu force-pushed the feat/table-reconciliation-utils branch from f759664 to 6dd4fca Compare July 17, 2025 06:41
Copy link
Collaborator

@fengjiachun fengjiachun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

@WenyXu WenyXu added this pull request to the merge queue Jul 17, 2025
Merged via the queue into GreptimeTeam:main with commit 8237646 Jul 17, 2025
76 of 77 checks passed
@WenyXu WenyXu deleted the feat/table-reconciliation-utils branch July 17, 2025 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-not-required This change does not impact docs. size/L
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants