HIVE-29459: [DR][HiveACIDReplication] Add clearDanglingTxnTask at the end#6334
Open
harshal-16 wants to merge 1 commit intoapache:masterfrom
Open
HIVE-29459: [DR][HiveACIDReplication] Add clearDanglingTxnTask at the end#6334harshal-16 wants to merge 1 commit intoapache:masterfrom
harshal-16 wants to merge 1 commit intoapache:masterfrom
Conversation
harshal-16
changed the title
… end
Details:
* Currently, at the end of replLoadTask, clearDanglingTxnTask is added. That works in normal scenario
```java
if (conf.getBoolVar(HiveConf.ConfVars.HIVE_REPL_CLEAR_DANGLING_TXNS_ON_TARGET)) { ClearDanglingTxnWork clearDanglingTxnWork = new ClearDanglingTxnWork(work.getDumpDirectory(), targetDb.getName());
Task<ClearDanglingTxnWork> clearDanglingTxnTask = TaskFactory.get(clearDanglingTxnWork, conf);
if (childTasks.isEmpty()) {
childTasks.add(clearDanglingTxnTask);
} else {
DAGTraversal.traverse(childTasks, new AddDependencyToLeaves(Collections.singletonList(clearDanglingTxnTask)));
}
} return 0;
```
* But if the no of events for incremental load is > hive.repl.approx.max.load.tasks
then Load operation can break down the tasks into batches of approx hive.repl.approx.max.load.tasks{{ (Not a hard limit)}}
* In this case, it can lead to pre-maturely cleaning of repl_txn_map and aborting the transaction in between the replication
because clearDanglingTxnTask gets called in between the batches rather than calling at the end only once per Load cycle.
Fix:
* Add clearDanglingTxnTask only if it doesn't have more work
Testing:
* Tested on live cluster
* Added test-case
harshal-16
force-pushed
the
HIVE-29459
branch
from
c2ac78c to
ed282ed
Compare
harshal-16
changed the title
asf-ci-hive
added
tests failed
tests pending
and removed
tests pending
tests failed
labels
|
Contributor
Author
|
Reported sonarqube issues are not from the changed files : https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6334&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
No
How was this patch tested?