Some Chiado snapshot files have changed

[yperbasis]

Got this email on 31 Mar:

[WARNING]: HASHES got CHANGED in snapshotter-bm-e3-chiado-n1

As a result of Erigon bug (or deliberated Erigon change) established files' hashes got changed in

snapshotter-bm-e3-chiado-n1

As a precautionary measure:
files were NOT published
a PR against erigon-snapshot was NOT raised

Please check the attached report for details and take your decision accordingly.

hashes_check_report.txt

[VBulikov]

yperbasis added Imp1 as Importance

[michelemodolo]

Granting admin access to you @mh0lt

[mh0lt]

I think that the root cause of this problem is comes from the way that this script: https://github.com/erigontech/scripts/blob/main/snapshots/snaprelease/snaprelease-remote.sh processes the erigon snapshots directory.

Erigon internally has the concept of visible and dirty files:

• Visble files are a clean view of all current files with those which are involved with background operations exluded this is the set of files which the rest of the erigon code uses for processing.
• Dirty files are all of the files currently in the directory, which includes some which are part processed by background operations.

Although https://github.com/erigontech/scripts/blob/main/snapshots/snaprelease/snaprelease-remote.sh stops erigon before processing snapshots for uploading this is not enough to guarantee a clean file set, because if erigon is stopped in the middle of background file operations it does not clean up its files, it leaves the process to be completed the next time it starts.

Some of the operations in these functions:
https://github.com/erigontech/scripts/blob/cba448aeadd21d0c6a621f79266976cea9030c92/snapshots/snaprelease/snaprelease-remote.sh#L295 https://github.com/erigontech/scripts/blob/cba448aeadd21d0c6a621f79266976cea9030c92/snapshots/snaprelease/snaprelease-remote.sh#L322, and all of the functions which do raw file processing are acting on the dirty file set.

This means in practice what we are distributing to the cloud are a set of dirty files which are in an undefined state.

I think in some cases these files are in an undefined removal state - which leads to the 0 size we see.

As erigon protects itself from the impact of dirty files - this does not impact the operating node it will just ignore these even if they are transmitted via the downloader. The actual downloader does not do this as it operates at a lower level of the process and acts just on files, but it seems that it may require some defensive code.

I think this leads to the following question:

Are we trying to deliver a clean visible file set across the network, or is it ok to deliver dirty files and expect all layers of the code to deal with it.

If it is the former I think we need to refactor https://github.com/erigontech/scripts/blob/cba448aeadd21d0c6a621f79266976cea9030c92/snapshots/snaprelease/snaprelease-remote.sh and the go utilities it uses to operate on visible files. If it is the latter we will need to go through the downloader and make sure it does the right thing when it gets files in an undefined state.

My preference would be to change the snaprelease-remote to operate on visible files.

A third option may be to make the closedown process of erigon more atomic so it leaves a clean directory - but I'm not sure how feasible that is.

[mh0lt]

It maybe that this is contributing to the issue and maybe a quicker fix:

https://github.com/erigontech/scripts/blob/cba448aeadd21d0c6a621f79266976cea9030c92/snapshots/snaprelease/snaprelease.sh#L257

[mh0lt]

Having looked at https://github.com/erigontech/scripts/blob/cba448aeadd21d0c6a621f79266976cea9030c92/snapshots/snaprelease/snaprelease-remote.sh some more it seems to me that all of the copying and status checking code in this script is operating on dirty files.

If we want to run this kind of thing at a script level it may be that we need a utility to create a view using sym links, but I suspect that this will confuse git.

It may be better to think about recoding this in go and using: https://github.com/go-git/go-git for the git processing. We can then script this to act on the visible file set.

[mh0lt]

This is the latest status for this:

We have now decided that the best fix is to adjust the snapshotter workflow to do the following:

if publishable fails when run after stopping erigon then run again".

erigon seg retire

If publishable fails again abort with an error.

Note: If erigon seg retire does not run fast enough in practice, there is an alternative fix in this branch which should be less process heavy: https://github.com/erigontech/erigon/tree/remove_overlap_cmd -if necessary we can fallback to that.

[michelemodolo]

publishable currently fails in case of:

• missing files (i.e. erigon hasn't produced a file yet)
• gaps within any snapshot

I think that erigon seg retire might help only in case of missing files so I think that it should be run only in that case.

However, even in case of missing files erigon seg retire sometimes doesn't help: for example we're having a missing file problem within the sepolia snapshotter for days and erigon seg retire didn't help.

So I'm wondering whether erigon seg retire would better be run as a standalone command if a missing file condition applies (without any certainty that it would help, as mentioned here above about the sepolia snapshotter's case), case by case, instead of including it as part of the snapshots automation (which would cause potential drawbacks like pipeline timeouts).
Nonetheless I'll include it to the snapshots automation on a specific branch so that some tests can be done.

FYI @AskAlexSharov