Augment Filecoin snapshots with F3 information

[LesnyRumcajs]

Summary

This tracks the initiative of augmenting regular Filecoin snapshots with F3 information to streamline syncing the F3 chain for all Filecoin clients.

Motivation

Tasks

• Implement the F3-augmented snapshots in Forest (note the forest.car.zst format - do we need to handle our index specially?).
  • Ensure snapshots are consumable by Forest/Lotus/Venus
• Communications if actions are required, e.g., on the ChainSafe infrastructure side.

Risks & Dependencies

We need to reach consensus within all implementation teams on the snapshot format (multiple snapshots/multiple roots/something else) and the data included. It'd be ideal if the impact on the snapshot service is minimal and the node handles the complexity.

Additional Links & Resources

Tracking filecoin-project/go-f3#480

[BigLep]

Doc discussed on 2025-05-13 at FDS: https://docs.google.com/document/d/1uVPlItYb3JZ2ZsAWLhlpE8EccNkdtn4MNtoRDJtLEok/edit?usp=sharing_eil&ts=6822f52b

Some notes I captured (not complete):

1. Optimize for sequential reads
2. It would be good for design doc, explain why we can't/don't use carv2 and why tar doesn't work.
3. There was an agreement to use something like TAR as a container for the existing snapshot data and certificate change.
4. Work items that were identified
   • Some publicly discoverable / readable doc defining the snapshot format. Lightweight FRC seems reasonable.
   • go-f3 adds support to export the certificate chain: Add API for exporting certificate chain for Filecoin snapshots filecoin-project/go-f3#480
   • Forest has the ability to generate this new "snapshot format"
     • Lotus will need to do this as well so that Forest CI can confirm Forest and Lotus generate the same result.
   • Forest/Lotus/Venus will need to be updated to import this new snapshot format.

Keys things we want to achieve:

1. Things just work for users. No extra steps.

[LesnyRumcajs]

Some remarks I noted afterwards, for completion:

• there shouldn't be any interface-breaking changes to the snapshot consumers - --import-snapshot and existing flags should contain any extra logic,
• this means the snapshot file must be a single one - maybe packed with tar, maybe a custom format that says that there are several CAR files included. In general, it should be straightforward to extract only one of the required pieces, e.g., only F3 and streaming should be easy to achieve.
• new format should allow for future changes (some versioning).
  it's fine if the format becomes backwards incompatible - we can do the change with a network upgrade so that we know that all nodes are up-to-date and able to handle new snapshots.
• The choice for design is essentially up-to-us, it's just required to not introduce any weird technologies that might not be supported everywhere (it needs to be supported at least with Rust and Golang).

[hanabi1224]

Design doc has been updated to address the feedbacks

[BigLep]

Design doc has been updated to address the feedbacks

@masih: this is the doc to review

[masih]

@LesnyRumcajs Can I get comment permissions on that doc please? thanks.