Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CAD] Add CAD custom indexer template #629

Merged
merged 55 commits into from
Sep 30, 2024
Merged
Show file tree
Hide file tree
Changes from 39 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
461d457
add custom indexer doc
0xaptosj Sep 24, 2024
e7f4477
Update _meta.tsx
0xaptosj Sep 24, 2024
c95c098
Update create-aptos-dapp.mdx
0xaptosj Sep 24, 2024
4778fb6
Update custom-indexer.mdx
0xaptosj Sep 24, 2024
c02a2ef
Merge branch 'main' into j/add-indexer-template-to-cad
0xaptosj Sep 24, 2024
2690c7f
Update custom-indexer.mdx
0xaptosj Sep 25, 2024
9c61527
Update custom-indexer.mdx
0xaptosj Sep 25, 2024
e287ffd
Update custom-indexer.mdx
0xaptosj Sep 25, 2024
9776f69
Update custom-indexer.mdx
0xaptosj Sep 25, 2024
2b56c12
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 25, 2024
c2ae489
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 25, 2024
a4ed6d5
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 25, 2024
7b93bf8
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 25, 2024
b871c11
add where to find connection string in neon
0xaptosj Sep 26, 2024
b97ab98
mention where to find sql queries
0xaptosj Sep 26, 2024
0278ec0
add install rust instructions
0xaptosj Sep 26, 2024
fed96e0
Merge branch 'main' into j/add-indexer-template-to-cad
0xaptosj Sep 26, 2024
af2413b
add demo
0xaptosj Sep 26, 2024
7ccf879
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
f324c37
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
a3c0265
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
c2a240d
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
5466459
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
de1d55a
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
12578b1
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
88cd75e
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
0991a7a
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
35f687e
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
8b81c47
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
a3ceb39
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
b5909ce
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
583b206
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
ef2f51f
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
03c295e
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
cab7061
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
c25274e
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
4f13473
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
8815e9e
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
c727d4c
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
e6b4643
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
5130b5b
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
57fefeb
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
52e0d7b
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
8b22e62
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
131983b
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
4676da1
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
9f9df55
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
79b2a3f
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
0ff87ce
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
10c0fe4
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
40bb372
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
33b21c3
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
4520f3a
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
8d6f1bf
Update apps/nextra/pages/en/build/create-aptos-dapp/templates/custom-…
0xaptosj Sep 26, 2024
8a8c994
Update custom-indexer.mdx
0xaptosj Sep 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion apps/nextra/pages/en/build/create-aptos-dapp.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,8 @@ The goals of the templates are to:
- [NFT minting dapp Template](create-aptos-dapp/templates/nft-minting-dapp.mdx)
- [Token minting dapp Template](create-aptos-dapp/templates/token-minting-dapp.mdx)
- [Token staking dapp Template](create-aptos-dapp/templates/token-staking-dapp.mdx)
- [Telegram Mini app Template](create-aptos-dapp/templates/clicker-game-tg-mini-app.mdx)
- [Custom indexer template](create-aptos-dapp/templates/custom-indexer.mdx)

## Tools `create-aptos-dapp` utilizes

Expand All @@ -90,4 +92,3 @@ The goals of the templates are to:
- Aptos TS SDK
- Aptos Wallet Adapter
- Node based Move commands

Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,7 @@ export default {
"token-staking-dapp": {
title: "Token staking dapp Template",
},
"custom-indexer": {
title: "Custom Indexer Template",
},
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,230 @@
---
title: "Create Aptos Dapp Custom Indexer Template"
---

import { Callout, Steps } from 'nextra/components';

# Create Aptos Dapp Custom Indexer Template

The custom indexer template provides a starter dapp with all the necessary infrastructure to build a full stack app with custom indexer support.

You can preview the template at: https://aptos-full-stack-demo.vercel.app/

The Custom Indexer template provides:

- Folder structure - A pre-made dapp folder structure with `src` for frontend, `contract` for Move contract and `indexer` for custom indexer.
- Dapp infrastructure - All required dependencies a dapp needs to start building on the Aptos network.
- Wallet Info implementation - Pre-made `WalletInfo` components to demonstrate how one can use to read a connected Wallet info.
- Message board functionality - Pre-made `MessageBoard` component to create, update and read messages from the Move smart contract.
- Analytics dashboard - A pre-made `Analytics` component to show the number of messages created and updated.
- Point program - A minimal example to show you how to define a point program based on events (e.g. create message, update message) and show that on the analytics dashboard, with sorting.

## Generate the custom indexer template

On your terminal, navigate to the directory you want to work in and run:
```bash
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
npx create-aptos-dapp@latest
```

Follow the CLI prompts.

## Getting started

<Steps>

### Publish the contract

Run the below command to publish the contract on-chain:
```bash
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
npm run move:publish
```
This command will:
1. Publish the contract to the chain.
2. Set the `MODULE_ADDRESS` in the `.env` file to set the contract object address.

### Create a postgres database

Sign up for [Neon postgres](https://neon.tech/) which is our cloud Postgres provider and create a new project. Find the connection string in Neon console's quickstart section, it should look like `postgresql://username:password@host/neondb?sslmode=require` and set it in the `.env` file as `DATABASE_URL`.

### Setup the database

<Callout>
Everything here should be run in the `indexer` folder.
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
</Callout>

Install Rust.

```sh
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```

Install diesel cli to run migrations.

```sh
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
cargo install diesel_cli --no-default-features --features postgres
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
```

Run all pending database migrations. This will create all the tables in the database.

```sh
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
diesel migration run \
--config-file="src/db_migrations/diesel.toml" \
--database-url="postgresql://username:password@neon_host/db_name?sslmode=require"
```

In case you want to revert all migrations to delete all tables because you want to re-index all data.

```sh
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
diesel migration revert \
--all \
--config-file="src/db_migrations/diesel.toml" \
--database-url="postgresql://username:password@neon_host/db_name?sslmode=require"
```

In case you want to make any change to the database, you can add a new migration.

```sh
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
diesel migration generate create-abc-table \
--config-file="src/db_migrations/diesel.toml"
```

### Sign up for Aptos Build

Sign up for [Aptos Build](https://developers.aptoslabs.com/), create a new project and get the API token.
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved

### Run the custom indexer locally

In the `indexer` folder, make a copy of `example.config.yaml` and rename it to `config.yaml`. Fill in the following:
- `starting_version`: The tx version (Aptos concept, similar to block height) from which you want to start indexing
- `postgres_connection_string`: The connection string of the postgres database, DO NOT include the `?sslmode=require` because diesel doesn't recognize it correctly, instead we handle it in the code
- `contract_address`: The address of the Move contract
- `auth_token`: Aptos Build API token

Start the custom indexer locally:

```sh
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
cargo run --release -- -c config.yaml
```

### Run the frontend

```bash
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
npm run dev
```

</Steps>

### Building the frontend

The template utilizes React with Next.js for the frontend, and it is styled with [Tailwind CSS](https://tailwindcss.com/) and [shadcn/ui](https://ui.shadcn.com/).
All dapp components should be added into the `components` folder and it is recommended to create a `app` folder to hold all future pages in your project.

### Writing a Move contract

The template comes with a `contract` folder that holds all Move smart contract related files. Under the `sources` folder you will find a `*.move` file with a super basic implementation of a Move module that stores a message and updates it. This is to help you get started with writing your own Smart Contract.

### Smart contract and frontend communication

For a frontend to submit a transaction to a smart contract, it needs to call an entry function. The boilerplate provides you with an `entry-functions` folder to hold all your dapp entry function requests.
Additionaly, for a frontend to fetch data, it reads the data from the databse. You can find the database queries in the `src/db` folder.

## Ready for Mainnet

If you started your dapp on testnet, and you are happy with your testing, you will want to get the dapp on mainnet.

To publish the smart contract on mainnet, we need to change some configuration.

Open the `.env` file and make the below changes:

> Note: Make sure you have created an existing account on Aptos `mainnet`

1. Create a new database or a new branch on Neon Postgres for production and update the `DATABASE_URL` in the `.env` file in your frontend.
2. Change the `APP_NETWORK` value to `mainnet`.
3. Update the `MODULE_PUBLISHER_ACCOUNT_ADDRESS` to be the existing account address.
4. Update the `MODULE_PUBLISHER_PRIVATE_KEY` to be the existing account private key.
5. Run `npm run move:publish` to publish your move module on Aptos mainnet.

## Deploy frontend to a live server

`create-aptos-dapp` provides an npm command to easily deploy the static site to [Vercel](https://vercel.com/home).

At the root of the folder, simply run:

```bash filename="Terminal"
npm run deploy
```

Then, follow the prompts. Please refer to Vercel docs to learn more about the [Vercel CLI](https://vercel.com/docs/cli)

If you are looking for different services to deploy the static site to, `create-aptos-dapp` utilizes Vite as the development tool, so you can follow the [Vite deployment guide](https://vitejs.dev/guide/static-deploy). In a nutshell, you would need to:

1. Run `npm run build` to build a static site
2. Run `npm run preview` to see how your dapp would look like on a live server
3. Next, all you need is to deploy your static site to a live server, there are some options for you to choose from and can follow [this guide](https://vitejs.dev/guide/static-deploy) on how to use each
Comment on lines +167 to +169
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd put . at the end of each of these lines to be consistent with the rest.


## Deploy indexer to a live server

We recommend using Google Cloud Run to host the indexer, Secret Manager to store `config.yaml` and Artifact Registry to store the indexer docker image.

### Build the docker image locally and run the container locally

Build the docker image targeting linux/amd64 as we will eventually push the image to Artifact Registry and deploy it to Cloud Run, which only supports linux/amd64.

```sh
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
docker build --platform linux/amd64 -t indexer .
```

You can run the docker container locally to make sure it works. Mac supports linux/amd64 emulation so you can run the x86 docker image on Mac.

```sh
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
docker run -p 8080:8080 -it indexer
```

### Push the locally build docker image to Artifact Registry

Login to google cloud

```sh
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
gcloud auth login
```

Create a repo in the container registry and push to it. You can learn more about publishing to Artifact Registry on their [docs](https://cloud.google.com/artifact-registry/docs/docker/pushing-and-pulling#pushing).

Authorize docker to push to Artifact Registry. Set the default region below to your region.

```sh
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
# update us-west2 to your region, you can find it in google cloud
gcloud auth configure-docker us-west2-docker.pkg.dev
```

Tag the docker image.

```sh
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
# update us-west2 to your region, you can find it in google cloud
docker tag indexer us-west2-docker.pkg.dev/google-cloud-project-id/repo-name/indexer
```

Push to the Artifact Registry.

```sh
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
# update us-west2 to your region, you can find it in google cloud
docker push us-west2-docker.pkg.dev/google-cloud-project-id/repo-name/indexer
```

### Upload the config.yaml file to Secret Manager

Go to secret manager and create a new secret using the `config.yaml` file. Please watch this video walkthrough carefully: https://drive.google.com/file/d/1bbwe617fqM31swqc9W5ck8G8eyg3H4H2/view

### Run the container on Cloud Run

Please watch this video walkthrough carefully and follow the exact same setup: https://drive.google.com/file/d/1JayWuH2qgnqOgzVuZm9MwKT42hj4z0JN/view.

Go to the Cloud Run dashboard, create a new service, then select the container image from Artifact Registry. Make sure to add a volume to load the config.yaml file from Secret Manager, then mount the volume to the container.

You can learn more about Cloud Run in their [docs](https://cloud.google.com/run/docs/quickstarts/deploy-container).

**NOTE**: Always allocate CPU so it always runs instead of only run when there is traffic. Min and max instances should be 1.
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved

## Re-indexing

**WARNING**: Do not ever try to backfill the data, as logic like point calculation is incremental. If you backfill (e.g. processing same event twice), you will get wrong point data. So please always revert all migrations and re-index from the first tx your contract deployed.
0xaptosj marked this conversation as resolved.
Show resolved Hide resolved
Loading