The Infra for w3protocol
A UCAN based API to for storing CARs and registering uploads, built on ucanto and SST.
The server-side implementation of the store/* and upload/* capabilities defined in w3protocol.
The repo contains the infra deployment code and the api implementation.
├── billing - usage accounting and reporting to the payment system
├── carpark - lambda for announce new CARs in the carpark bucket
├── filecoin - lambdas to get content into Filecoin deals
├── replicator - lambda to replicate buckets to R2
├── satnav - lambda to listen for new CARs and write CAR indexes in satnav
├── stacks - sst and aws cdk code to deploy all the things
├── ucan-invocation - kinesis log data stream and its lambda consumers
└── upload-api - lambda & dynamoDB implementation of the upload-api http gateway
To work on this codebase you need:
- Node.js >= v16 (prod env is node v16)
- Install the deps with
npm i
You can then run the tests locally with npm test.
To try out a change submit a PR and you'll get temporary infra rolled out for you automatically at https://<pr#>.up.web3.storage.
sst is the framework we use to define what to deploy. Read the docs! https://sst.dev
Deployments are managed by seed.run.
The main branch is deployed to https://staging.up.web3.storage and staging builds are promoted to prod manually via the UI at https://console.seed.run
You can use sst to create a custom dev deployment on aws, with a local dev console for debugging.
To do that you need
- An AWS account with the AWS CLI configured locally
- Copy
.env.tplto.env.local
Then run npm start to deploy dev services to your aws account and start dev console
npm startSee: https://docs.sst.dev for more info on how things get deployed.
To test the Stripe integration, set the STRIPE_SECRET_KEY and STRIPE_ENDPOINT_SECRET
secrets using sst secret set (use npm exec sst -- secret set to do this in the root of this project).
STRIPE_SECRET_KEY should be set to the "secret" API key found on the test mode API keys page: https://dashboard.stripe.com/test/apikeys
To get a value for STRIPE_ENDPOINT_SECRET you'll need to create a webhook on https://dashboard.stripe.com/test/webhooks and point it at the Stripe webhook handler for your development server. You can get webhook handler URL by adding /stripe to the end of the
w3infra-UploadApiStack ApiEndpoint output after running npm start and letting it deploy.
The full value of STRIPE_ENDPOINT_SECRET will look something like https://z1jsa5b24d.execute-api.us-west-2.amazonaws.com/stripe.
You can use the stripe CLI to trigger test events, like:
stripe trigger checkout.session.completed
To run per-package tests, first install Docker Desktop (https://www.docker.com/) and ensure it is running.
Next, ensure the AWS_REGION, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are set in your terminal. They do
not need to be set to real values - the following works in bash-like shells:
export AWS_REGION='us-west-2'; export AWS_ACCESS_KEY_ID='NOSUCH'; export AWS_SECRET_ACCESS_KEY='NOSUCH'
Finally, to run the tests for all packages, run:
npm test
Or to run the tests for a single package, run:
npm test -w <path/to/package>
Integration tests run by default on post-deploy stage of seed.run deployment pipeline. These integration tests will run with the deployed infrastructure for the given stages (PR / staging).
It is also possible to run the integration tests with a development deploy of sst. For this, you can run:
npm run deploy
npm run test-integration
Please notice that appropriate environment variables must be set for the development deploy. For this you are required to setup your .env.local file as instructed in Getting started section. You can read more here on how to setup CI with a w3account.
Ensure the following variables are set in the env when deploying
The root domain to deploy the w3up API to. e.g up.web3.storage. The value should match a hosted zone configured in route53 that your aws account has access to.
The domain to deploy the roundabout API to. e.g roundabout.web3.storage. The value should match a hosted zone configured in route53 that your aws account has access to.
URL of the w3access service.
DID of the filecoin aggregator service.
URL of the filecoin aggregator service.
DID of the content claims service.
URL of the content claims service.
DID of the filecoin deal tracker service.
URL of the filecoin deal tracker service.
DID of the upload-api ucanto server. e.g. did:web:up.web3.storage. Optional: if omitted, a did:key will be derrived from PRIVATE_KEY
Access key for S3 like cloud object storage to replicate content into.
Secret access key for S3 like cloud object storage to replicate content into.
Endpoint for S3 like cloud object storage to replicate content into.
Bucket name to replicate written CAR files.
Bucket name to replicate written .idx files.
Bucket name to replicate root CID to car CIDs mapping.
Bucket name where delegations are stored.
A comma-separated string of ServiceDIDs in use.
Data source name for Sentry application monitoring service.
AWS ARN for Elastic IPFS SQS indexer used to request Elastic IPFS to index given CAR files.
AWS URL for Elastic IPFS SQS indexer used to request Elastic IPFS to index given CAR files.
AWS ARN for Elastic IPFS SQS multihashes used to enqueue multihashes for indexed CAR files.
AWS ARN for Elastic IPFS DynamoDB table used to store blocks and positions for indexed CAR files.
Postmark API token, which is used by the email verification system to send emails.
API token for Mailslurp, which is used in integration tests. To invalidate or refresh tokens, head to the Mailslurp dashboard.
Set production secrets in aws SSM via sst secrets. The region must be set to the one you deploy that stage to
# set `PRIVATE_KEY` for prod
$ npx sst secrets set --region us-west-2 --stage prod PRIVATE_KEY "MgCblCY...="To set a fallback value for staging or an ephmeral PR build use sst secrets set-fallback
# set `PRIVATE_KEY` for any stage in us-east-2
$ npx sst secrets set-fallback --region us-east-2 PRIVATE_KEY "MgCZG7...="note The fallback value can only be inherited by stages deployed in the same AWS account and region.
Confirm the secret value using sst secrets list
$ npx sst secrets list --region us-east-2
PRIVATE_KEY MgCZG7...= (fallback)
$ npx sst secrets list --region us-west-2 --stage prod
PRIVATE_KEY M...=The multibase encoded ED25519 keypair used as the signing key for the upload-api.
Generated by @ucanto/principal EdSigner via ucan-key
Example: MgCZG7EvaA...1pX9as=
The base64pad multibase encoded ED25519 keypair used as the signing key for content-claims.
Generated by @ucanto/principal EdSigner via ucan-key
Example: MgCZG7EvaA...1pX9as=
The HTTP Basic auth token for the UCAN Invocation entrypoint, where UCAN invocations can be stored and proxied to the UCAN Stream.
Example: MgCZG7EvaA...1pX9as=
Stripe API key for reporting per customer usage.
A UCAN based [RPC] API over HTTP.
The RPC endpoint for invoking UCAN cababilities. Supported abilities are defined below in UCAN Capabilities
The POST body must contain a CAR encoded UCAN.
Content-Type: application/car header must be present in the the request headers.
An endpoint for receiving signed Stripe webhooks.
Returns version info for this api in JSON
{ "name": "@web3-storage/upload-api", "did": "did:foo:bar", "version": "3.0.0", "commit": "sha1", "branch": "main" }Implements store/* and upload/* capabilities defined in w3protocol
Register a CAR CID to be stored. Returns an S3 compatible signed upload URL usable for that CAR.
Source: api/service/store/add.js
List the CAR CIDs for the issuer.
Source: api/service/store/list.js
Remove a CAR by CAR CID.
Source: api/service/upoload/remove.js
Source: api/service/upload/add.js
Source: api/service/upload/list.js
Source: api/service/upload/remove.js
Use the JS upload-client to handle the details of content-addressing your files, encoding them into a CAR, and sending it to the service with a valid UCAN.
import { Agent } from '@web3-storage/access'
import { store } from '@web3-storage/capabilities/store'
import { upload } from '@web3-storage/capabilities/upload'
import { uploadFile } from '@web3-storage/upload-client'
// holds your identity on this device
const agent = await Agent.create()
// your upload... either from a <input type=file> or from a path on your fs using `files-from-path`
const file = new Blob(['Hello World!'])
// the Content-Address for your file, derived client side before sending to the service.
// Returns once your data is safely stored.
const cid = await uploadFile({
issuer: agent.issuer,
with: agent.currentSpace(),
proofs: await agent.proofs([store, upload]),
}, file)