DAOS-19616 dtx: fix race issues between DTX resync and rebuild - b28 by Nasf-Fan · Pull Request #19227 · daos-stack/daos

Nasf-Fan · 2026-10-09T08:26:26Z

On each target, before scanning the container shard for rebuild,
the logic will wait its global stable epoch (per container based)
to exceed rebuild epoch. It will guarantee that all the inflight
IO result that are triggered before rebuild start is visible to
the subsequent rebuild logic whether on DTX leader or not.
Before handling some modification on DTX leader, need to check
whether the epoch for current modification is newer than pool's
rebuild epoch or not. If not, then return -DER_UPDATE_AGAIN to
the client that will retry the modification sometime later.
In theory, it is possible that when exclude some engine, related
ULT maybe trying to abort a DTX because of some failure happened
during local modification (but related modification has succeeded
on remote targets). Related DTX_ABORT RPC has been sent out but
not been received or processed by related remote targets. Under
such case, the DTX leader is switched because of related engine
is exlcuded. And then DTX resync is triggered by new DTX leader
who may regard such DTX as committable since all alive targets
claim 'PREPARED'. Then the new DTX leader will make different
decision that conflict with old DTX leader did. That may cause
related DTX to be aborted on some targets but committed on the
others, as to data inconsistency among related redundancy group.
The new logic forbids to abort DTX from old DTX leader to avoid
such inconsistency.
If the modification is in rebuilding/reintegrating, then related
modification will use synchronous DTX to refresh related global
stable epoch more quickly.
During DTX resync, the DTX CHECK RPC should not be sent to the
target that will be reintegrated (with PO_COMP_ST_UP). Because
such target may have removed related DTX entries.

Signed-off-by: Fan Yong fan.yong@hpe.com
Co-authored-by: Xuezhao Liu xuezhao.liu@hpe.com
Co-authored-by: Wang Shilong shilong.wang@hpe.com

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2026-10-09T08:26:40Z

Ticket title is 'FIO crc mismatch detected on 4x node IB config while running HSN cable pulls test'
Status is 'Awaiting backport'
Job should run at elevated priority (1)
https://daosio.atlassian.net/browse/DAOS-19616

daosbuild3 · 2026-10-09T10:21:18Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19227/3/execution/node/1355/log

daosbuild3 · 2026-10-09T12:01:05Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19227/5/execution/node/1355/log

daosbuild3 · 2026-10-09T14:04:25Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19227/6/execution/node/1361/log

daosbuild3 · 2026-10-10T05:24:34Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19227/11/execution/node/1356/log

gnailzenh · 2026-10-10T08:24:26Z

+		 * For old backend, we cannot use the container's global stable epoch to defense
+		 * potential inflight IO. Then directly wait for 1 minute on the first container.
+		 */
+		if (rpt->rt_remaining_wait_time > 0) {


I assume this container wouldn't be checked for twice, so we don't need to check/set rt_remaining_wait_time at all for 2.6 container?

2.6 does not support global stable epoch, so need such logic for 2.6

daosbuild3 · 2026-10-10T09:15:46Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19227/15/execution/node/1350/log

daosbuild3 · 2026-10-10T10:22:08Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19227/16/display/redirect

gnailzenh · 2026-10-11T03:44:31Z

 				continue;

-			/* Skip non-healthy one. */
-			if (target->ta_comp.co_status != PO_COMP_ST_UP &&


why UP should be removed from here?

It is not removed, still be there at L199.

gnailzenh · 2026-10-11T03:53:56Z

-	DSHR_CORRUPT		= 5,
+	DSHR_NEED_COMMIT = DTX_ST_MAX + 100,
+	DSHR_NEED_RETRY,
+	DSHR_IGNORE,


style: I think it's confusing (and easy to make mistake) for vos_dtx_check() to return two different sets of value, would be better to add a new output paramter to vos_dtx_check(), but we can do this cleanup later on master

gnailzenh · 2026-10-11T04:15:48Z

-
 int
-dtx_abort(struct ds_cont_child *cont, struct dtx_entry *dte, daos_epoch_t epoch)
+dtx_abort(struct ds_cont_child *cont, struct dtx_entry *dte, daos_epoch_t epoch, uint32_t ver)


I'm trying to under the ver parameter, it's current pool map version and can be different with dte::dte_ver right? Why is zero passed in some cases?

Yes, the pool map version can be changed according to the latest pool map. The dte::dte_ver is the version when the DTX is generated. Zero version means that the caller wants to abort related entry by force.

gnailzenh · 2026-10-11T04:38:45Z

 		return 0;
 	}

+again:


why do we close/open again after sleep?

It is possible that someone may destroy the container during our sleep for wait. Then refreshing global stable epoch for such container will be stopped. If we do not put the reference, we may wait/block there until timeout.

gnailzenh · 2026-10-11T04:50:38Z

-			goto out;
-		}
-	}
-


Is it safe to remove this?

I think so. Need @liuxuezhao to confirm.

daosbuild3 · 2026-10-11T09:43:41Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19227/25/execution/node/1629/log

daosbuild3 · 2026-10-11T14:52:53Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19227/25/execution/node/1734/log

1. On each target, before scanning the container shard for rebuild, the logic will wait its global stable epoch (per container based) to exceed rebuild epoch. It will guarantee that all the inflight IO result that are triggered before rebuild start is visible to the subsequent rebuild logic whether on DTX leader or not. 2. Before handling some modification on DTX leader, need to check whether the epoch for current modification is newer than pool's rebuild epoch or not. If not, then return -DER_UPDATE_AGAIN to the client that will retry the modification sometime later. 3. In theory, it is possible that when exclude some engine, related ULT maybe trying to abort a DTX because of some failure happened during local modification (but related modification has succeeded on remote targets). Related DTX_ABORT RPC has been sent out but not been received or processed by related remote targets. Under such case, the DTX leader is switched because of related engine is exlcuded. And then DTX resync is triggered by new DTX leader who may regard such DTX as committable since all alive targets claim 'PREPARED'. Then the new DTX leader will make different decision that conflict with old DTX leader did. That may cause related DTX to be aborted on some targets but committed on the others, as to data inconsistency among related redundancy group. The new logic forbids to abort DTX from old DTX leader to avoid such inconsistency. 4. If the modification is in rebuilding/reintegrating, then related modification will use synchronous DTX to refresh related global stable epoch more quickly. 5. During DTX resync, the DTX CHECK RPC should not be sent to the target that will be reintegrated (with PO_COMP_ST_UP). Because such target may have removed related DTX entries. Add server side environment variable DAOS_REBUILD_CONTINUE_ON_TIMEOUT to control whether continue rebuild process or not if got timeout when wait for global stable epoch. It is false by default and rebuild will failout (and restart) when got timeout. If explicitly set it as true, then there is risk of data lost in subsequent rebuild process if race with inflight IO. It is a temporary solution. Signed-off-by: Fan Yong <fan.yong@hpe.com> Co-authored-by: Xuezhao Liu <xuezhao.liu@hpe.com> Co-authored-by: Wang Shilong <shilong.wang@hpe.com>

daosbuild3 · 2026-10-11T16:27:13Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19227/26/execution/node/1356/log

wangshilong · 2026-10-12T01:19:17Z

 		D_GOTO(close, rc);
 	}

+	rc = vos_cont_get_global_stable_epoch(coh, &gse);


probably skip wait for reclaim/fail_reclaim.

github-actions Bot added the priority Ticket has high priority (automatically managed) label Oct 9, 2026

Nasf-Fan force-pushed the Nasf-Fan/DAOS-19616_b28 branch 3 times, most recently from 57eeaa1 to 92d88df Compare October 9, 2026 10:19

Nasf-Fan force-pushed the Nasf-Fan/DAOS-19616_b28 branch 2 times, most recently from cb9634f to fdba1f9 Compare October 9, 2026 11:59

Nasf-Fan force-pushed the Nasf-Fan/DAOS-19616_b28 branch from fdba1f9 to acdd437 Compare October 9, 2026 14:02

Nasf-Fan force-pushed the Nasf-Fan/DAOS-19616_b28 branch 5 times, most recently from 4978f8d to 701593c Compare October 10, 2026 05:22

Nasf-Fan force-pushed the Nasf-Fan/DAOS-19616_b28 branch 3 times, most recently from cad90f8 to 2869b3b Compare October 10, 2026 06:56

gnailzenh previously approved these changes Oct 10, 2026

View reviewed changes

Nasf-Fan dismissed gnailzenh’s stale review via b1a2157 October 10, 2026 09:14

Nasf-Fan force-pushed the Nasf-Fan/DAOS-19616_b28 branch from 2869b3b to b1a2157 Compare October 10, 2026 09:14

Nasf-Fan force-pushed the Nasf-Fan/DAOS-19616_b28 branch from b1a2157 to b589d74 Compare October 10, 2026 10:20

Nasf-Fan force-pushed the Nasf-Fan/DAOS-19616_b28 branch 4 times, most recently from 3617998 to 9ebdc6f Compare October 10, 2026 14:31

Nasf-Fan changed the title ~~DAOS-19616 dtx: fix some race issues between DTX resync and rebuild~~ DAOS-19616 dtx: fix race issues between DTX resync and rebuild - b28 Oct 10, 2026

Nasf-Fan requested review from gnailzenh, liuxuezhao and wangshilong October 10, 2026 14:43

daltonbohning mentioned this pull request Oct 10, 2026

DAOS-19632 object: combined patch for test - 2 #19235

Draft

6 tasks

Nasf-Fan force-pushed the Nasf-Fan/DAOS-19616_b28 branch 4 times, most recently from 4c60248 to 27b7deb Compare October 11, 2026 04:14

gnailzenh reviewed Oct 11, 2026

View reviewed changes

Nasf-Fan force-pushed the Nasf-Fan/DAOS-19616_b28 branch from 27b7deb to 40b2780 Compare October 11, 2026 14:50

Nasf-Fan force-pushed the Nasf-Fan/DAOS-19616_b28 branch from 40b2780 to 9600127 Compare October 11, 2026 16:25

wangshilong reviewed Oct 12, 2026

View reviewed changes

Conversation

Nasf-Fan commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Oct 9, 2026

Uh oh!

daosbuild3 commented Oct 9, 2026

Uh oh!

daosbuild3 commented Oct 9, 2026

Uh oh!

daosbuild3 commented Oct 9, 2026

Uh oh!

daosbuild3 commented Oct 10, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Oct 10, 2026

Uh oh!

daosbuild3 commented Oct 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

daosbuild3 commented Oct 11, 2026

Uh oh!

daosbuild3 commented Oct 11, 2026

Uh oh!

daosbuild3 commented Oct 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

Nasf-Fan commented Oct 9, 2026 •

edited

Loading