Admin/XMover: Add CrateDB shard analyzer and movement tool #523
Conversation
WalkthroughAdds XMover: a new administrative toolkit for CrateDB that analyzes shard distribution, recommends and optionally executes safe shard relocations, validates moves, monitors recoveries, exposes a Click CLI and script entrypoint, provides models/utilities/analysis/operational modules, extensive docs, and unit tests. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor User
participant CLI as xmover CLI
participant Client as CrateDBClient
participant Analyzer as ShardAnalyzer
participant Recommender as ShardRelocationRecommender
participant DB as CrateDB
User->>CLI: xmover recommend [options]
CLI->>Client: init + test_connection()
CLI->>Analyzer: init(client)
CLI->>Recommender: execute(constraints, auto_execute, validate, dry_run)
Recommender->>Analyzer: generate_rebalancing_recommendations(constraints)
Analyzer->>Client: get_nodes_info()/get_shards_info()
Client-->>Analyzer: nodes, shards
Analyzer-->>Recommender: recommendations
alt auto_execute and not dry_run
Recommender->>Client: execute_query(ALTER TABLE ... REROUTE MOVE SHARD ...)
Client->>DB: POST /_sql
DB-->>Client: result
Client-->>Recommender: success/failure
Recommender->>Recommender: _wait_for_recovery_capacity()
else dry_run
Recommender-->>CLI: render recommendations (no execution)
end
CLI-->>User: rendered output / SQL / status
sequenceDiagram
autonumber
actor User
participant CLI as xmover monitor-recovery
participant Client as CrateDBClient
participant Monitor as RecoveryMonitor
participant DB as CrateDB
User->>CLI: xmover monitor-recovery [--watch]
CLI->>Monitor: start(watch)
loop refresh_interval (watch mode)
Monitor->>Client: get_all_recovering_shards(filters)
Client->>DB: query sys.allocations/sys.shards
DB-->>Client: rows
Client-->>Monitor: RecoveryInfo[]
Monitor-->>CLI: formatted table + deltas
end
CLI-->>User: summary
sequenceDiagram
autonumber
actor User
participant CLI as xmover validate-move
participant Analyzer as ShardAnalyzer
participant Client as CrateDBClient
User->>CLI: xmover validate-move <schema.table> <shard> <from> <to>
CLI->>Analyzer: init(client)
CLI->>Analyzer: validate_move_safety(recommendation, max_disk_usage)
Analyzer->>Client: lookups (nodes/shards)
Client-->>Analyzer: details
Analyzer-->>CLI: (is_safe, reason)
CLI-->>User: verdict + SQL command
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Suggested reviewers
Poem
✨ Finishing touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
amotl
force-pushed
the
xmover
branch
2 times, most recently
from
6a6b8f0 to
5068671
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
amotl
force-pushed
the
xmover
branch
2 times, most recently
from
dac2a6d to
cb05b4d
Compare
amotl
force-pushed
the
xmover
branch
4 times, most recently
from
2c6ef7f to
21dd237
Compare
|
Thank you, @WalBeh! |
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as off-topic.
This comment was marked as off-topic.
There was a problem hiding this comment.
All code is not taking any possible partitions into account which can lead to serious issues.
Also, I'm a bit concerned about all the more complex recommendation and specially, movement logic, as most logic is implemented in CrateDB itself and can work counter-productive. E.g. if you move some shard because of X, CrateDB will move it elsewhere caused by it's only logic.
Honestly I feel (and sorry, I really appreciate all the effort), that this tool should be trimmed down a lot to do very basic movements, like exchanging primaries with replicas if there is an unbalance.
Otherwise, I'd only keep the reporting.
| self._zone_conflict_cache: Dict[Tuple[str, int, str], Union[str, None]] = {} | ||
| self._node_lookup_cache: Dict[str, Union[NodeInfo, None]] = {} | ||
| self._target_nodes_cache: Dict[Tuple[float, frozenset[Any], float, float], List[NodeInfo]] = {} | ||
| self._cache_hits = 0 | ||
| self._cache_misses = 0 |
There was a problem hiding this comment.
Do we need caching at all? Isn't this more an initial bootstrapping of required data structures? I don't see any cache invalidation. This increases complexity quite a bit, I'd suggest to avoid any unnecessary complexity as this increases review and maintenance overhead.
| mean_count = sum(counts) / len(counts) | ||
| if mean_count == 0: | ||
| return 100.0 |
There was a problem hiding this comment.
Isn't this the average and not the mean? https://en.wikipedia.org/wiki/Median
| return 100.0 | ||
|
|
||
| # Calculate coefficient of variation | ||
| variance = sum((count - mean_count) ** 2 for count in counts) / len(counts) |
There was a problem hiding this comment.
Why not using the builtin statistics.variance?
| table_name: Optional[str] = None, | ||
| min_size_gb: Optional[float] = None, | ||
| max_size_gb: Optional[float] = None, | ||
| for_analysis: bool = False, |
There was a problem hiding this comment.
| for_analysis: bool = False, | |
| only_started: bool = False, |
| for_analysis: If True, includes all shards regardless of state (for cluster analysis) | ||
| If False, only includes healthy shards suitable for operations |
There was a problem hiding this comment.
| for_analysis: If True, includes all shards regardless of state (for cluster analysis) | |
| If False, only includes healthy shards suitable for operations | |
| started_only: If False, includes all shards regardless of state (for cluster analysis) | |
| If True, only includes started shards suitable for operations |
| query = """ | ||
| SELECT | ||
| s.table_name, | ||
| s.schema_name, |
There was a problem hiding this comment.
| s.schema_name, | |
| s.schema_name, | |
| s.partition_ident, |
| query = """ | ||
| SELECT node['name'] as node_name | ||
| FROM sys.shards | ||
| WHERE schema_name = ? AND table_name = ? AND id = ? |
| """Information about a shard""" | ||
|
|
||
| table_name: str | ||
| schema_name: str |
There was a problem hiding this comment.
| schema_name: str | |
| schema_name: str | |
| partition_ident: str |
| """Recommendation for moving a shard""" | ||
|
|
||
| table_name: str | ||
| schema_name: str |
There was a problem hiding this comment.
| schema_name: str | |
| schema_name: str | |
| partition_ident: str |
| ### Advanced Troubleshooting | ||
| ```bash | ||
| # Validate specific moves before execution | ||
| xmover validate-move SCHEMA.TABLE SHARD_ID FROM_NODE TO_NODE |
There was a problem hiding this comment.
should take partition into account
About
@WalBeh contributed a powerful utility that uses CrateDB's system tables to find out about cluster imbalances related to shard number and shard size distribution across the whole cluster. Thanks!
After analysing the situation, the program presents solutions in form of SQL commands to bring the cluster into a more balanced state again.
Install
uv pip install --upgrade 'cratedb-toolkit @ git+https://github.com/crate/cratedb-toolkit.git@xmover'Documentation
https://cratedb-toolkit--523.org.readthedocs.build/admin/xmover/
References