fix(nayduck) - fix flaky state_sync_routed test

[wacban]

The test was failing when the node was slow to catch up and was still at height 0 when entering this loop:

nearcore/pytest/tests/sanity/state_sync_routed.py

Lines 106 to 107 in ba1ed9f

	if catch_up_height >= observed_height:
	break

In this case the hash_ here would be the genesis block hash:

nearcore/pytest/tests/sanity/state_sync_routed.py

Lines 118 to 120 in ba1ed9f

	if ctx.get_balances() == ctx.expected_balances:
	ctx.send_moar_txs(hash_, 3, False)
	logger.info(f"Sending moar txs at height {boot_height}")

The ctx.send_moar_txs would send it to a node that is fully caught it and has already garbage collected the genesis block. In this case the transaction would be invalid and the balance check would fail. This would in turn lead to the test being stuck forever in the final loop:

nearcore/pytest/tests/sanity/state_sync_routed.py

Lines 142 to 149 in ba1ed9f

	while ctx.get_balances() != ctx.expected_balances:
	assert time.time() - started < TIMEOUT
	logger.info(
	"Waiting for the old node to catch up. Current balances: %s; Expected balances: %s"
	% (ctx.get_balances(), ctx.expected_balances))
	time.sleep(1)
	logger.info("Old node caught up to expected balances: %s" %
	ctx.expected_balances)

This PR contains a few fixes and improvement:

• Fixing the issue described above by using the boot_nodes head's block hash when creating transactions.
• Decreasing the timeout of the test to 100s + 2s per block to make it fail faster (5min instead of 20min)
• Always execute the "catch up" loop at least once to make the test more interesting.
• More assertions and logs to fail faster
• Removed the pool_time overwrite (now the default 0.25 is used). It doesn't make any difference but the code is nicer and shorter.
• Random mini refactorings.

[Longarithm]

LGTM, thank you!

Let's also change timeout in pytest --timeout=600 sanity/state_sync_routed.py manytx 115, etc. to something more reasonable.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.46%. Comparing base (ba1ed9f) to head (383bd74).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #12821      +/-   ##
==========================================
- Coverage   70.47%   70.46%   -0.02%     
==========================================
  Files         847      847              
  Lines      175003   175003              
  Branches   175003   175003              
==========================================
- Hits       123329   123311      -18     
- Misses      46426    46439      +13     
- Partials     5248     5253       +5     

Flag	Coverage Δ
backward-compatibility	0.16% <ø> (ø)
db-migration	0.16% <ø> (ø)
genesis-check	1.41% <ø> (ø)
linux	70.07% <ø> (-0.01%)	⬇️
linux-nightly	70.09% <ø> (+0.01%)	⬆️
pytests	1.70% <ø> (ø)
sanity-checks	1.52% <ø> (ø)
unittests	70.29% <ø> (-0.02%)	⬇️
upgradability	0.20% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.