[qs] tolerate block timestamp being updated asynchronously

[zekun000]

In previous commit #14757, we move timestamp update to async callback to ensure the consistency between mempool, qs and consensus. However it introduces the race of timestamp can be updated by sync_to first then async callback and violates the assertion.

[trunk-io]

⏱️ 1h 3m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
test-target-determinator	10m	🟩 🟩
execution-performance / test-target-determinator	9m	🟩 🟩
check	7m	🟩 🟩
rust-doc-tests	5m	🟩
rust-cargo-deny	5m	🟩 🟩 🟩
rust-doc-tests	5m	🟩
rust-targeted-unit-tests	5m	⬜
check-dynamic-deps	4m	🟩 🟩 🟩
fetch-last-released-docker-image-tag	3m	🟩 🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
general-lints	1m	🟩 🟩 🟩
semgrep/ci	1m	🟩 🟩 🟩
Backport PR	35s	🟥 🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
test-target-determinator	5m	4m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite realistic_env_max_load success on bfdabe3699c325952725a6d5cbbe4ef53f0a357c

two traffics test: inner traffic : committed: 13189.26 txn/s, latency: 3014.46 ms, (p50: 3000 ms, p70: 3000, p90: 3300 ms, p99: 6300 ms), latency samples: 5014840
two traffics test : committed: 100.04 txn/s, latency: 2561.82 ms, (p50: 2400 ms, p70: 2500, p90: 2700 ms, p99: 8400 ms), latency samples: 1820
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.245, avg: 0.221", "QsPosToProposal: max: 0.523, avg: 0.384", "ConsensusProposalToOrdered: max: 0.327, avg: 0.310", "ConsensusOrderedToCommit: max: 0.504, avg: 0.467", "ConsensusProposalToCommit: max: 0.804, avg: 0.776"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.17s no progress at version 2347275 (avg 0.22s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 7.70s no progress at version 2347273 (avg 7.70s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on b6c01b9ab025d778fa74270795d7b0cb9bf95342 ==> bfdabe3699c325952725a6d5cbbe4ef53f0a357c

Compatibility test results for b6c01b9ab025d778fa74270795d7b0cb9bf95342 ==> bfdabe3699c325952725a6d5cbbe4ef53f0a357c (PR)
1. Check liveness of validators at old version: b6c01b9ab025d778fa74270795d7b0cb9bf95342
compatibility::simple-validator-upgrade::liveness-check : committed: 12389.77 txn/s, latency: 2328.97 ms, (p50: 1900 ms, p70: 2100, p90: 2300 ms, p99: 20400 ms), latency samples: 493080
2. Upgrading first Validator to new version: bfdabe3699c325952725a6d5cbbe4ef53f0a357c
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7112.50 txn/s, latency: 3998.69 ms, (p50: 4400 ms, p70: 4700, p90: 4900 ms, p99: 5200 ms), latency samples: 138040
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6198.56 txn/s, latency: 5202.94 ms, (p50: 5300 ms, p70: 5500, p90: 7400 ms, p99: 7700 ms), latency samples: 233420
3. Upgrading rest of first batch to new version: bfdabe3699c325952725a6d5cbbe4ef53f0a357c
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6194.20 txn/s, latency: 4545.61 ms, (p50: 5100 ms, p70: 5300, p90: 5900 ms, p99: 6200 ms), latency samples: 122700
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6740.83 txn/s, latency: 4734.35 ms, (p50: 4800 ms, p70: 5100, p90: 6800 ms, p99: 7200 ms), latency samples: 228800
4. upgrading second batch to new version: bfdabe3699c325952725a6d5cbbe4ef53f0a357c
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11429.46 txn/s, latency: 2388.47 ms, (p50: 2600 ms, p70: 2700, p90: 2900 ms, p99: 3000 ms), latency samples: 201440
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10688.75 txn/s, latency: 2907.07 ms, (p50: 2800 ms, p70: 2900, p90: 4200 ms, p99: 5500 ms), latency samples: 348340
5. check swarm health
Compatibility test for b6c01b9ab025d778fa74270795d7b0cb9bf95342 ==> bfdabe3699c325952725a6d5cbbe4ef53f0a357c passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on b6c01b9ab025d778fa74270795d7b0cb9bf95342 ==> bfdabe3699c325952725a6d5cbbe4ef53f0a357c

Compatibility test results for b6c01b9ab025d778fa74270795d7b0cb9bf95342 ==> bfdabe3699c325952725a6d5cbbe4ef53f0a357c (PR)
Upgrade the nodes to version: bfdabe3699c325952725a6d5cbbe4ef53f0a357c
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1227.07 txn/s, submitted: 1230.05 txn/s, failed submission: 2.99 txn/s, expired: 2.99 txn/s, latency: 2573.02 ms, (p50: 2400 ms, p70: 2700, p90: 4200 ms, p99: 6300 ms), latency samples: 106720
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1216.01 txn/s, submitted: 1218.30 txn/s, failed submission: 2.29 txn/s, expired: 2.29 txn/s, latency: 2587.45 ms, (p50: 2400 ms, p70: 2900, p90: 3600 ms, p99: 5300 ms), latency samples: 106420
5. check swarm health
Compatibility test for b6c01b9ab025d778fa74270795d7b0cb9bf95342 ==> bfdabe3699c325952725a6d5cbbe4ef53f0a357c passed
Upgrade the remaining nodes to version: bfdabe3699c325952725a6d5cbbe4ef53f0a357c
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1201.43 txn/s, submitted: 1203.49 txn/s, failed submission: 2.06 txn/s, expired: 2.06 txn/s, latency: 2575.61 ms, (p50: 2400 ms, p70: 2700, p90: 4300 ms, p99: 6600 ms), latency samples: 104880
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

💚 All backports created successfully

Status	Branch	Result
✅	aptos-release-v1.21

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details