block files: long read consistency of `View`

[AskAlexSharov]

Technically View - it's ReadOnlyTransaction (analog from DBMS's) - and we need: high isolation level - which guarantee reads consistency for long time (between several blockReader.GetBlockByNumber calls).

so, we need "snapshot-isolation isolation level" :-)
https://en.wikipedia.org/wiki/Isolation_(database_systems)
https://en.wikipedia.org/wiki/Snapshot_isolation

E3 already providing this functionality on state files. See Aggregator.BeginFilesRo + Aggregator.Close (each time RPCDaemon does tx := db.Begin under the hood it cals Aggregator.BeginFilesRo.

Corner-cases to consider:

• erigon may start on empty datadir - and it will download files
• erigon may restart in the middle of download - means it will start on full datadir but will continue download
• erigon may restart in the middle of BuildMissedIndices - means it will start on "complete download" dir but not enough .idx files yet
• erigon may restart after download and buildMissedIndices done - but with --no-downloader flag - means it may not know "is download finish?"
• rpcdaemon may start without erigon (at all) on existing datadir - and it must immediately start serving RPC requests (it can't be blocked by request to erigon about "did you finish downloading?"). Now we solved this problem by storing in db list of "good files" - but it's not the best thing.

See also: #11318 (comment)

Sub-tasks:

• block files: split _dirtyFiles and _visibleFiles  #11417
• block files: keep merged files in sub-dir and mv after merge of all types is done #11430
• [ ]

==================
WARNING: DATA RACE
Read at 0x00c080da3e68 by goroutine 1911:
  github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks.(*BlockReader).HeaderByHash()
      /home/ubuntu/erigon/turbo/snapshotsync/freezeblocks/block_reader.go:416 +0x2d6
  github.com/erigontech/erigon/eth/protocols/eth.AnswerGetBlockHeadersQuery()
      /home/ubuntu/erigon/eth/protocols/eth/handlers.go:58 +0xfd4
  github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).getBlockHeaders66.func1()
      /home/ubuntu/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:640 +0xc4
  github.com/erigontech/erigon-lib/kv/temporal.(*DB).View()
      /home/ubuntu/erigon/erigon-lib/kv/temporal/kv_temporal.go:103 +0x18d
  github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).getBlockHeaders66()
      /home/ubuntu/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:639 +0x39a
  github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).handleInboundMessage()
      /home/ubuntu/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:802 +0x13b
  github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).HandleInboundMessage()
      /home/ubuntu/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:773 +0x128
  github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).HandleInboundMessage-fm()
      <autogenerated>:1 +0x6d
  github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.pumpStreamLoop[go.shape.*uint8].func2()
      /home/ubuntu/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:247 +0x162

Previous write at 0x00c080da3e68 by goroutine 133:
  github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks.(*Segment).closeIdx()
      /home/ubuntu/erigon/turbo/snapshotsync/freezeblocks/block_snapshots.go:154 +0x358
  github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks.(*RoSnapshots).buildMissedIndices.func2()
      /home/ubuntu/erigon/turbo/snapshotsync/freezeblocks/block_snapshots.go:843 +0x31c
  github.com/tidwall/btree.(*Map[go.shape.int,go.shape.*uint8]).nodeScan()
      /home/ubuntu/go/pkg/mod/github.com/tidwall/btree@v1.6.0/map.go:279 +0x2f8
  github.com/tidwall/btree.(*Map[go.shape.int,go.shape.*uint8]).scan()
      /home/ubuntu/go/pkg/mod/github.com/tidwall/btree@v1.6.0/map.go:270 +0xa4
  github.com/tidwall/btree.(*Map[go.shape.int,go.shape.*uint8]).Scan()
      /home/ubuntu/go/pkg/mod/github.com/tidwall/btree@v1.6.0/map.go:259 +0x2b
  github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks.(*RoSnapshots).buildMissedIndices()
      /home/ubuntu/erigon/turbo/snapshotsync/freezeblocks/block_snapshots.go:835 +0x6b7
  github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks.(*RoSnapshots).buildMissedIndicesIfNeed()
      /home/ubuntu/erigon/turbo/snapshotsync/freezeblocks/block_snapshots.go:743 +0x21a
  github.com/erigontech/erigon/turbo/snapshotsync/freezeblocks.(*BlockRetire).BuildMissedIndicesIfNeed()
      /home/ubuntu/erigon/turbo/snapshotsync/freezeblocks/block_snapshots.go:1544 +0x1a7
  github.com/erigontech/erigon/eth/stagedsync.DownloadAndIndexSnapshotsIfNeed()
      /home/ubuntu/erigon/eth/stagedsync/stage_snapshots.go:284 +0x159e
  github.com/erigontech/erigon/eth/stagedsync.SpawnStageSnapshots()
      /home/ubuntu/erigon/eth/stagedsync/stage_snapshots.go:175 +0x1ce
  github.com/erigontech/erigon/eth/stagedsync.DefaultStages.func1()
      /home/ubuntu/erigon/eth/stagedsync/default_stages.go:49 +0x152
  github.com/erigontech/erigon/eth/stagedsync.(*Sync).runStage()
      /home/ubuntu/erigon/eth/stagedsync/sync.go:529 +0x285
  github.com/erigontech/erigon/eth/stagedsync.(*Sync).Run()
      /home/ubuntu/erigon/eth/stagedsync/sync.go:413 +0x593
  github.com/erigontech/erigon/turbo/stages.ProcessFrozenBlocks()
      /home/ubuntu/erigon/turbo/stages/stageloop.go:143 +0xcf
  github.com/erigontech/erigon/turbo/stages.StageLoop()
      /home/ubuntu/erigon/turbo/stages/stageloop.go:78 +0x116
  github.com/erigontech/erigon/eth.(*Ethereum).Start.gowrap2()
      /home/ubuntu/erigon/eth/backend.go:1559 +0x144

Goroutine 1911 (running) created at:
  github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.pumpStreamLoop[go.shape.*uint8]()
      /home/ubuntu/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:241 +0x4aa
  github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.SentryReconnectAndPumpStreamLoop[go.shape.*uint8]()
      /home/ubuntu/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:196 +0x8e7
  github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).RecvUploadHeadersMessageLoop()
      /home/ubuntu/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:114 +0x297
  github.com/erigontech/erigon/p2p/sentry/sentry_multi_client.(*MultiClient).StartStreamLoops.gowrap3()
      /home/ubuntu/erigon/p2p/sentry/sentry_multi_client/sentry_multi_client.go:81 +0x6e

Goroutine 133 (running) created at:
  github.com/erigontech/erigon/eth.(*Ethereum).Start()
      /home/ubuntu/erigon/eth/backend.go:1559 +0xe1e
  github.com/erigontech/erigon/node.(*Node).Start()
      /home/ubuntu/erigon/node/node.go:129 +0x3b3
  github.com/erigontech/erigon/node.StartNode()
      /home/ubuntu/erigon/node/node.go:419 +0x2e
  github.com/erigontech/erigon/turbo/node.(*ErigonNode).run()
      /home/ubuntu/erigon/turbo/node/node.go:69 +0x9b
  github.com/erigontech/erigon/turbo/node.(*ErigonNode).Serve()
      /home/ubuntu/erigon/turbo/node/node.go:49 +0x86
  main.runErigon()
      /home/ubuntu/erigon/cmd/erigon/main.go:103 +0x685
  github.com/erigontech/erigon/turbo/app.MakeApp.func1()
      /home/ubuntu/erigon/turbo/app/make_app.go:71 +0x17e
  github.com/urfave/cli/v2.(*Command).Run()
      /home/ubuntu/go/pkg/mod/github.com/urfave/cli/v2@v2.27.2/command.go:276 +0x1578
  github.com/urfave/cli/v2.(*App).RunContext()
      /home/ubuntu/go/pkg/mod/github.com/urfave/cli/v2@v2.27.2/app.go:333 +0x1274
  github.com/urfave/cli/v2.(*App).Run()
      /home/ubuntu/go/pkg/mod/github.com/urfave/cli/v2@v2.27.2/app.go:307 +0xc8
  main.main()
      /home/ubuntu/erigon/cmd/erigon/main.go:51 +0x8d
==================