Update minion task metadata ZNode path#8959
Conversation
Codecov Report
@@ Coverage Diff @@
## master #8959 +/- ##
=============================================
- Coverage 69.70% 26.66% -43.04%
+ Complexity 4709 1 -4708
=============================================
Files 1817 1806 -11
Lines 94791 94534 -257
Branches 14178 14149 -29
=============================================
- Hits 66073 25207 -40866
- Misses 24090 66926 +42836
+ Partials 4628 2401 -2227
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
Jackie-Jiang
left a comment
There was a problem hiding this comment.
Ideally, we want to automatically move the ZNRecord from the old path to the new path. When reading the ZK metadata, if we find the old path exist:
- If the new one also exist, remove the old one
- If the new one does not exist, copy the old record to the new path, then remove the old one
pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java
Outdated
Show resolved
Hide resolved
pinot-common/src/main/java/org/apache/pinot/common/minion/MinionTaskMetadataUtils.java
Outdated
Show resolved
Hide resolved
pinot-common/src/main/java/org/apache/pinot/common/minion/MinionTaskMetadataUtils.java
Outdated
Show resolved
Hide resolved
We can do 1 safely. I'd prefer we keep the current logic and discuss how to implement the proposal correctly before going that direction. |
...mmon/src/main/java/org/apache/pinot/common/minion/RealtimeToOfflineSegmentsTaskMetadata.java
Outdated
Show resolved
Hide resolved
...ot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskGenerator.java
Outdated
Show resolved
Hide resolved
npawar
left a comment
There was a problem hiding this comment.
minor comment. lgtm otherwise
pinot-common/src/main/java/org/apache/pinot/common/metadata/ZKMetadataProvider.java
Outdated
Show resolved
Hide resolved
664cc08 to
e800c2a
Compare
What does this PR do?
This PR updates minion task metadata ZNode path from
/MINION_TASK_METADATA/${taskType}/${tableNameWithType}to/MINION_TASK_METADATA/${tableNameWithType}/${taskType}.The logic is backward compatible: if there exists minion task metadata in ZNode path
/MINION_TASK_METADATA/${tableNameWithType}/${taskType}, it keeps using it; otherwise, it uses the new ZNode path.How was the code tested?
/MINION_TASK_METADATA/${taskType}/${tableNameWithType}, and (2) there does not exists minion task metadata.