Skip to content

DAOS-19616 dtx: fix race issues for DTX resync - #19240

Open
Nasf-Fan wants to merge 1 commit into
masterfrom
Nasf-Fan/DAOS-19616_2
Open

Nasf-Fan wants to merge 1 commit into
masterfrom
Nasf-Fan/DAOS-19616_2

Conversation

@Nasf-Fan

Copy link
Copy Markdown
Contributor
  1. In theory, it is possible that when exclude some engine, related ULT maybe trying to abort a DTX because of some failure happened during local modification (but related modification has succeeded on remote targets). Related DTX_ABORT RPC has been sent out but not been received or processed by related remote targets. Under such case, the DTX leader is switched because of related engine is exlcuded. And then DTX resync is triggered by new DTX leader who may regard such DTX as committable since all alive targets claim 'PREPARED'. Then the new DTX leader will make different decision that conflict with old DTX leader did. That may cause related DTX to be aborted on some targets but committed on the others, as to data inconsistency among related redundancy group. The new logic forbids to abort DTX from old DTX leader to avoid such inconsistency.

  2. During DTX resync, the DTX CHECK RPC should not be sent to the target that will be reintegrated (with PO_COMP_ST_UP). Because such target may have removed related DTX entries.

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

@github-actions

Copy link
Copy Markdown

Ticket title is 'FIO crc mismatch detected on 4x node IB config while running HSN cable pulls test'
Status is 'Awaiting backport'
Job should run at elevated priority (1)
https://daosio.atlassian.net/browse/DAOS-19616

@github-actions github-actions Bot added the priority Ticket has high priority (automatically managed) label Oct 11, 2026
@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19240/1/execution/node/1808/log

1. In theory, it is possible that when exclude some engine, related
   ULT maybe trying to abort a DTX because of some failure happened
   during local modification (but related modification has succeeded
   on remote targets). Related DTX_ABORT RPC has been sent out but
   not been received or processed by related remote targets. Under
   such case, the DTX leader is switched because of related engine
   is exlcuded. And then DTX resync is triggered by new DTX leader
   who may regard such DTX as committable since all alive targets
   claim 'PREPARED'. Then the new DTX leader will make different
   decision that conflict with old DTX leader did. That may cause
   related DTX to be aborted on some targets but committed on the
   others, as to data inconsistency among related redundancy group.
   The new logic forbids to abort DTX from old DTX leader to avoid
   such inconsistency.

2. During DTX resync, the DTX CHECK RPC should not be sent to the
   target that will be reintegrated (with PO_COMP_ST_UP). Because
   such target may have removed related DTX entries.

Signed-off-by: Fan Yong <fan.yong@hpe.com>
@Nasf-Fan
Nasf-Fan force-pushed the Nasf-Fan/DAOS-19616_2 branch from 04e9c27 to 2514142 Compare October 11, 2026 16:48
@Nasf-Fan
Nasf-Fan marked this pull request as ready for review October 12, 2026 13:39
@Nasf-Fan
Nasf-Fan requested review from a team as code owners October 12, 2026 13:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority Ticket has high priority (automatically managed)

Development

Successfully merging this pull request may close these issues.

2 participants