[mempool] move the qs callback to be after mempool

[zekun000]

this commit guarantees the garbage collection order of mempool -> quorum store -> block store.

[trunk-io]

⏱️ 2h 10m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
forge-e2e-test / forge	41m	🟩 🟩 🟩
forge-compat-test / forge	21m	🟩
forge-framework-upgrade-test / forge	18m	🟩
test-target-determinator	13m	🟩 🟩 🟩
check-dynamic-deps	8m	🟩 🟩 🟩 🟩
rust-cargo-deny	6m	🟩 🟩 🟩
rust-doc-tests	5m	🟩
execution-performance / test-target-determinator	4m	🟩
check	4m	🟩
general-lints	3m	🟩 🟩 🟩
rust-move-tests	2m	🟩
semgrep/ci	2m	🟩 🟩 🟩 🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
file_change_determinator	45s	⬜ 🟩 🟩 🟩
file_change_determinator	33s	🟩 🟩 🟩
permission-check	15s	🟩 🟩 🟩 🟩
file_change_determinator	14s	🟩
permission-check	11s	🟩 🟩 🟩 🟩
execution-performance / single-node-performance	10s	🟩
permission-check	10s	🟩 🟩 🟩 🟩
Backport PR	9s	🟥 🟥
permission-check	8s	🟩 🟩 🟩
permission-check	7s	🟩 🟩
determine-docker-build-metadata	6s	🟩 🟩 🟩
permission-check	5s	🟩 🟩
rust-move-tests	1s	⬜

🚨 2 jobs on the last run were significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
general-lints	26s	2m
execution-performance / single-node-performance	10s	18m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite realistic_env_max_load success on 39f4570af2ee1dc78386236d3dd82add28a2d98e

two traffics test: inner traffic : committed: 13985.59 txn/s, submitted: 13986.17 txn/s, expired: 0.58 txn/s, latency: 2838.84 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3600 ms), latency samples: 5317620
two traffics test : committed: 99.97 txn/s, latency: 1625.80 ms, (p50: 1500 ms, p70: 1600, p90: 1700 ms, p99: 8200 ms), latency samples: 1660
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.270, avg: 0.236", "QsPosToProposal: max: 1.158, avg: 1.101", "ConsensusProposalToOrdered: max: 0.343, avg: 0.301", "ConsensusOrderedToCommit: max: 0.446, avg: 0.420", "ConsensusProposalToCommit: max: 0.743, avg: 0.720"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.87s no progress at version 2936802 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.44s no progress at version 2936800 (avg 8.44s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on 25a081116546670e62ca927ba90478de78557056 ==> 39f4570af2ee1dc78386236d3dd82add28a2d98e

Compatibility test results for 25a081116546670e62ca927ba90478de78557056 ==> 39f4570af2ee1dc78386236d3dd82add28a2d98e (PR)
Upgrade the nodes to version: 39f4570af2ee1dc78386236d3dd82add28a2d98e
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1054.87 txn/s, submitted: 1057.08 txn/s, failed submission: 2.21 txn/s, expired: 2.21 txn/s, latency: 2917.06 ms, (p50: 2400 ms, p70: 3000, p90: 5200 ms, p99: 8100 ms), latency samples: 95260
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1054.47 txn/s, submitted: 1055.57 txn/s, failed submission: 1.11 txn/s, expired: 1.11 txn/s, latency: 2846.24 ms, (p50: 2400 ms, p70: 2800, p90: 5500 ms, p99: 6900 ms), latency samples: 95400
5. check swarm health
Compatibility test for 25a081116546670e62ca927ba90478de78557056 ==> 39f4570af2ee1dc78386236d3dd82add28a2d98e passed
Upgrade the remaining nodes to version: 39f4570af2ee1dc78386236d3dd82add28a2d98e
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1048.46 txn/s, submitted: 1051.21 txn/s, failed submission: 2.75 txn/s, expired: 2.75 txn/s, latency: 2918.21 ms, (p50: 2400 ms, p70: 2700, p90: 6000 ms, p99: 8200 ms), latency samples: 91560
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on 25a081116546670e62ca927ba90478de78557056 ==> 39f4570af2ee1dc78386236d3dd82add28a2d98e

Compatibility test results for 25a081116546670e62ca927ba90478de78557056 ==> 39f4570af2ee1dc78386236d3dd82add28a2d98e (PR)
1. Check liveness of validators at old version: 25a081116546670e62ca927ba90478de78557056
compatibility::simple-validator-upgrade::liveness-check : committed: 13930.30 txn/s, latency: 2083.72 ms, (p50: 1800 ms, p70: 2000, p90: 2800 ms, p99: 4800 ms), latency samples: 532740
2. Upgrading first Validator to new version: 39f4570af2ee1dc78386236d3dd82add28a2d98e
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7152.50 txn/s, latency: 3750.68 ms, (p50: 4200 ms, p70: 4700, p90: 5000 ms, p99: 5200 ms), latency samples: 127900
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7368.09 txn/s, latency: 4306.02 ms, (p50: 4500 ms, p70: 4600, p90: 6300 ms, p99: 6700 ms), latency samples: 245520
3. Upgrading rest of first batch to new version: 39f4570af2ee1dc78386236d3dd82add28a2d98e
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7706.77 txn/s, latency: 3679.90 ms, (p50: 4100 ms, p70: 4400, p90: 4500 ms, p99: 4600 ms), latency samples: 140880
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7295.11 txn/s, latency: 4035.24 ms, (p50: 4300 ms, p70: 4400, p90: 4400 ms, p99: 5400 ms), latency samples: 273280
4. upgrading second batch to new version: 39f4570af2ee1dc78386236d3dd82add28a2d98e
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11741.17 txn/s, latency: 2284.59 ms, (p50: 2400 ms, p70: 2600, p90: 2800 ms, p99: 2900 ms), latency samples: 206460
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10335.01 txn/s, latency: 2967.09 ms, (p50: 2600 ms, p70: 2900, p90: 6000 ms, p99: 7400 ms), latency samples: 335300
5. check swarm health
Compatibility test for 25a081116546670e62ca927ba90478de78557056 ==> 39f4570af2ee1dc78386236d3dd82add28a2d98e passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking