Project Yellowstone: Old Faithful is the project to make all of Solana's history accessible, content addressable and available via a variety of means. The goal of this project is to completely replace the Bigtable dependency for Solana history access with a self-hostable, decentralized history archive that is usable for infrastructure providers, individual Solana users, explorers, indexers, and anyone else in need of historical access.
The core of the project is history archives in Content Addressable format (overview, specs). These represent a verifiable, immutable view of the Solana history. The CAR files that this project generates follows a schema specifically developed for Solana's historical archives.
The content addressable nature means that each epoch, block, transaction and shredding is uniquely identified by a content hash. By knowing this content hash you will be able to retreive a specific object of your interest. Retrievals can be made via IPFS, the Filecoin network, or even by hosting the CAR files yourself on disk, a ceph cluster, S3, you name it.
Indexes will be needed to map Solana's block numbers, transaction signatures and addresses to their respective CIDs. These indexes will be developed as part of this project.
The archive is currently under development. There will be two main ways to access the archive during its development phase:
- Via Filecoin: Through collaboration with Protocol Labs and a Filplus allocation we are uploading all historical data to Filecoin. From here, any user can access the full histortical archive verifiably and freely from the Filecoin network. This is helpful as a way to test retreivals and fetch individual transactions.
- Bulk transfer: If you want to start testing full epoch archives, we can provide you with access to a storage bucket from where you can pull the epoch archives.
The tooling used in this project is based in the radiance tool developed by Jump's Firedancer team. It is rapidely developing, and active development for this project is currently based out of this repository and branch: Radiance Triton.
The radiance tool utilises the rocksdb snapshots that have been generated by Warehouse nodes. From these snapshots a CAR file per epoch is generated. This CAR file then needs to be processed by Filecoin tools such as split-and-commp which generates the details needed for making a Filecoin deal.
Currently, this tool is being tested from the following warehouse archives:
- Solana Foundation (public)
- gs://mainnet-beta-ledger-us-ny5
- gs://mainnet-beta-ledger-europe-fr2
- gs://mainnet-beta-ledger-asia-sg1
- Triton One (private)
If you have warehouse nodes generating rocksdb archive snapshots, please contact lk@triton.one (even if they can't be made publicly available). We would like to have you generate CAR files for verification purposes.
Using the rocksdb archives, the Radiance tool can be used to generate one CAR file per epoch. This CAR file is then made available via storage providers such as Filecoin and private storage buckets.
CAR file generation produces a CAR containing a DAG. This DAG is reproducible and follows the structure of Epoch -> Block -> Transaction see schema. The CAR file generation is deterministic, so even if you use different rocksdb source snapshots you should end up with the same CAR output. This allows comparison between different providers.
The data generation flow is illustrated below:
We will be regularly updating the CIDs uploaded to Filecoin here so that you can test retrievals and the IPLD schema parsing. If you have questions, comments or concerns please feel free to open issues on this repo.
These are most useful for testing with. You can download these using lassie (see below) to access the transaction data:
| Signature | Epoch | Block | CID |
|---|---|---|---|
| 39V8tR2Q8Ar3WwMBfVTRPFr7AakLHy5wp7skJNBL7ET6ARoikqc1TaMiuXEtHiNPLQKoeiVr5XnKH8QtjdonN4yM | 0 | 1 | bafyreiafcknvju54cilmqkopps4d3t4n3azjhrweyouxfltbgdo7mbpuoy |
| 4hPVA21e1KLQsEQkpSHk1UfkBBUfotajpkqESuV4tgqEdtEyDufaczAdZzSLexLhjytczDdSUFgwCTancgUWFzym | 0 | 1 | bafyreig3utb4u6x2agmyugntbxph4uesiodzq5k7typduhppcf62sdgjiu |
| FAWA66fudpiwdRDDQ4DRxdJsRvawvauwg4vQkm98ZHFpXmW5N7xzRiTRpt8QiZ19s1aVbzKgXW6kEZanwHeDFNS | 0 | 1 | bafyreigd7wopxlspljqkumzyiechhb43dwz6f67n4zbh7gef3kro4dj7qa |
| 58A6RFEk5AoFqXTtKfqLhLrs7mhJYWGQTZegoCXPZuneM6Spi47SNYk2M6d9MVzHbC9CpBVk5vrq24yyNgeQNK2p | 0 | 1 | bafyreieysdgkid4p3hszup7mwvyji2gzlsrkx5rmuqdz2i2ujs3wyfxdby |
These will be fairly large downloads, so for testing please use the Transaction CIDs above.
| Epoch | CID |
|---|---|
| 0 | bafyreiagdtlc3xwhbeywzpwmxvwkogcujhlsm6f4cfdgpjpyu77gkubro4 |
Currently, querying can be done from Filecoin using the lassie utility:
lassie fetch <cid>
Once you have downlaoded the CID you can unpack it or use the dump2 tool available in the Radiance repo. This tool can help you review the contents returned from Lassie.
lassie fetch -o - <cid> | dump2 -
Other query methods that are on the roadmap:
- A wrapper for Filecoin retrievals that allows parsing of the retrieved schema/data
- A JSON RPC gateway that takes Solana JSONRPC requests and can respond to them via backends such as Filecoin or S3.
This project is currently managed by Triton One. If you want more information contact us via Telegram.
The originator of this project was Richard Patel (Twitter).
@immaterial.ink (Twitter) is currently the lead dev on this project at Triton One.
This work has been supported greatly by Protocol Labs (special shout out to anjor (Twitter) for all the guidance in Filecoin land to us Solana locals).
The Solana Foundation is funding this effort through a project grant.
Solana.fm was, alongside Richard and Triton, one of the initiators of this project.
Also thanks to all RPC providers and others who have (and are) providing input to and support for this process.