[Fix][Broker] Fix race condition between timeout and completion in OpAddEntry

[mattisonchao]

Motivation

In #4455 and #10740, The OpAddEntry try to using opAddEntry.addOpCount to ensure the opAddEntry we got is what we want.(avoid recycle cause race condition)

It seems exists a case causing another race condition. consider the case as follow:

Precondition

• We have an OpAddEntry that name is A.
• We got two threads: a timeout check thread and another is a write thread that wants to complete and recycle this OpAddEntry.
• Relative code is as follow(timeout check logic):

pulsar/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java

Lines 3825 to 3840 in caf7648

	private void checkAddTimeout() {
	long timeoutSec = config.getAddEntryTimeoutSeconds();
	if (timeoutSec < 1) {
	return;
	}
	OpAddEntry opAddEntry = pendingAddEntries.peek();
	if (opAddEntry != null) {
	boolean isTimedOut = opAddEntry.lastInitTime != -1
	&& TimeUnit.NANOSECONDS.toSeconds(System.nanoTime() - opAddEntry.lastInitTime) >= timeoutSec;
	if (isTimedOut) {
	log.error("Failed to add entry for ledger {} in time-out {} sec",
	(opAddEntry.ledger != null ? opAddEntry.ledger.getId() : -1), timeoutSec);
	opAddEntry.handleAddTimeoutFailure(opAddEntry.ledger, opAddEntry.addOpCount);
	}
	}
	}

Race condition process

1. When the timeout checker thread gets OpAddEntry A and continues to run until line 3835.
2. At the same time, the writing thread also gets OpAddEntryA, completes and recycles.
3. After step 2, OpAddEntryA is taken out of the recycling pool by other threads as OpAddEntryB. But the timeout thread has passed the timeout check(line 3834), and addOpCount is also obtained from OpAddEntry B, which will cause the compareAndSet check to pass.

Affect

• The timeout checker will make a new OpAddEntry complete timeout.

Modifications

• Record opAddCount before timeout judgment.

Verifying this change

• Make sure that the change passes the CI checks.

Documentation

• no-need-doc

[lhotari]

Good catch!

btw. previous revisit to the solution was in #10740

[Jason918]

LGTM

[michaeljmarshall]

LGTM, and thank you for the great analysis @mattisonchao!

[nicoloboschi]

@lhotari this pull must be ported to branch-2.10 also, thanks