Data ingestion for Tableland Basin
Tableland Basin is a secure and verifiable open data platform. The Basin Provider is a collection of daemons and libraries that work in conjuction with basin-cli
and basin-storage
.
basin-provider
is responsible for ingesting Basin Publications that are archived on Filecoin by basin-storage
. Publication data can be ingested in two forms:
- Continuously published data from a connected database (currently, only PostgreSQL is supported)
- Raw parquet files
See basin-cli
for usage info.
This repo contains the following crates:
basin_worker
: Daemon that exposes a Cap’n Proto RPC API for creating publications and ingesting publication data. [
basin-storage`](https://github.com/tablelandnetwork/basin-storage.git).basin_evm
: Rust bindings for thebasin-storage
EVM contract.basin_protocol
: Cap’n Proto protocol definitions and Rust bindings.basin_common
: Shared types and utilities.
🚧 Basin is currently not in a production-ready state. Any data that is pushed to the network may be subject to deletion. 🚧
basin_worker --help
Ingest daemon for Tableland Basin
Usage: basin_worker [OPTIONS] --export-bucket <EXPORT_BUCKET> --export-credentials <EXPORT_CREDENTIALS> --database-url <DATABASE_URL>
Options:
--evm-type <EVM_TYPE>
EVM type (other EVM flags are ignored when this is 'mem')
[env: EVM_TYPE=]
[default: remote]
Possible values:
- mem: Use an in-memory ephemeral EVM w/ random wallet (a BasinStorage contract will be deployed)
- remote: Requires wallet private key, contract address, provider URL, and chain ID
--evm-wallet-pk <EVM_WALLET_PK>
Wallet private key (ECDSA, secp256k1) to use with the EVM (must have PUB_ADMIN_ROLE)
[env: EVM_WALLET_PK=]
--evm-contract-address <EVM_CONTRACT_ADDRESS>
BasinStorage EVM contract address (ECDSA, secp256k1)
[env: EVM_CONTRACT_ADDRESS=]
--evm-provider-url <EVM_PROVIDER_URL>
EVM provider URL
[env: EVM_PROVIDER_URL=]
[default: http://127.0.0.1:8545]
--evm-chain-id <EVM_CHAIN_ID>
EVM chain ID
[env: EVM_CHAIN_ID=]
[default: 31337]
--export-bucket <EXPORT_BUCKET>
Parquet export GCS bucket
[env: EXPORT_BUCKET=]
--export-credentials <EXPORT_CREDENTIALS>
Parquet export sink credentials
[env: EXPORT_CREDENTIALS=]
--database-url <DATABASE_URL>
Postgres-style database URL
[env: DATABASE_URL=]
--bind-address <BIND_ADDRESS>
Host and port to bind the RPC API to
[env: BIND_ADDRESS=]
[default: 127.0.0.1:3000]
--bind-health-address <BIND_HEALTH_ADDRESS>
Host and port to bind the Health API to
[env: BIND_HEALTH_ADDRESS=]
[default: 127.0.0.1:3001]
-v, --verbosity...
Logging verbosity (repeat for more verbose logging)
[env: VERBOSITY=]
-q, --quiet
Silence logging
[env: QUIET=]
-h, --help
Print help (see a summary with '-h')
-V, --version
Print version
docker compose up
docker compose up crdb
cargo test
PRs accepted.
Small note: If editing the README, please conform to the standard-readme specification.
MIT AND Apache-2.0, © 2021-2023 Tableland Network Contributors