[Misc] Small improvements to forge, consensus observer and state sync.

[JoshLind]

Description

This PR offers several small improvements (each in their own commit):

1. Improve the FullNodeRebootStressTest by preventing node reboots from happening too close to the end of the test.
2. Disable consensus observer on VFNs (by default). VFNs are still opt-in, and will be turned on once a few improvements are made.
3. Bump the max message size for state sync (from 8MB to 10MB). This should help favour output syncing a little more.

Testing Plan

Existing test infrastructure.

[trunk-io]

⏱️ 11h 15m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
indexer-grpc-e2e-tests / test-indexer-grpc-docker-compose	8h 1m	⬜ 🟥
execution-performance / single-node-performance	1h 20m	🟩 🟥 🟥 🟥
test-target-determinator	13m	🟩 🟩 🟩
execution-performance / test-target-determinator	12m	🟩 🟩 🟩
check	11m	🟩 🟩 🟩
rust-move-tests	11m	🟩
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-move-tests	10m	🟩
rust-cargo-deny	8m	🟩 🟩 🟩 🟩
rust-doc-tests	6m	🟩
rust-doc-tests	5m	🟩
rust-doc-tests	5m	🟩
fetch-last-released-docker-image-tag	5m	🟩 🟩 🟩
check-dynamic-deps	5m	🟩 🟩 🟩 🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite realistic_env_max_load success on 4137739855f1e03badf35cda361c760f7fa6b3f5

two traffics test: inner traffic : committed: 11708.30 txn/s, latency: 3398.77 ms, (p50: 3000 ms, p70: 3300, p90: 4200 ms, p99: 7200 ms), latency samples: 4451800
two traffics test : committed: 100.04 txn/s, latency: 2874.87 ms, (p50: 2600 ms, p70: 2900, p90: 3900 ms, p99: 6700 ms), latency samples: 1720
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.247, avg: 0.231", "QsPosToProposal: max: 0.452, avg: 0.366", "ConsensusProposalToOrdered: max: 0.335, avg: 0.323", "ConsensusOrderedToCommit: max: 0.575, avg: 0.530", "ConsensusProposalToCommit: max: 0.902, avg: 0.853"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.50s no progress at version 1860615 (avg 0.23s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 7.36s no progress at version 1860613 (avg 7.36s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

❌ Forge suite framework_upgrade failure on 628e88b8a1971b4986dfb2b88ec763090f85c82f ==> 4137739855f1e03badf35cda361c760f7fa6b3f5

Forge test runner terminated:
Trailing Log Lines:
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0               1/1     Running     0          18m
aptos-node-1-validator-0               1/1     Running     0          18m
aptos-node-2-validator-0               1/1     Running     0          18m
aptos-node-3-validator-0               1/1     Running     0          18m
genesis-aptos-genesis-eforge75-mdsmd   0/1     Completed   0          21m

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

❌ Forge suite compat failure on 628e88b8a1971b4986dfb2b88ec763090f85c82f ==> 4137739855f1e03badf35cda361c760f7fa6b3f5

Forge test runner terminated:
Trailing Log Lines:
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          18m
aptos-node-1-validator-0                1/1     Running     0          18m
aptos-node-2-validator-0                1/1     Running     0          18m
aptos-node-3-validator-0                1/1     Running     0          18m
genesis-aptos-genesis-eforge238-95kpn   0/1     Completed   0          21m

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking