Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(store-sync): add util to fetch snapshot from indexer with SQL API #2996

Merged
merged 32 commits into from
Sep 18, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
8b76bdd
feat(store-sync): add util to fetch snapshot from dozer
alvrs Jul 31, 2024
52dd46b
add missing changes
alvrs Jul 31, 2024
121479d
skip test
alvrs Jul 31, 2024
9abfffc
parallelize sql query requests
alvrs Jul 31, 2024
367daec
add KeySchema
alvrs Aug 1, 2024
dc154d2
move protocol-parser changes to store-sync
alvrs Aug 1, 2024
bf885cc
move KeySchema
alvrs Aug 1, 2024
aca18b9
keep zustand types
alvrs Aug 1, 2024
b36d6fe
rename things
alvrs Aug 1, 2024
175fe7c
fix type error
alvrs Aug 1, 2024
a677f32
missed one PartialTable
alvrs Aug 1, 2024
9c398d8
update logic in dozer/getSnapshot
alvrs Aug 1, 2024
f22747a
chore: include main branch when fetching during pre-release action
Kooshaba Aug 5, 2024
0af11fa
docs(state-sync/dozer): first version (#3033)
qbzzt Aug 22, 2024
fa2bbc7
Merge branch 'main' into alvrs/dozer-query
alvrs Sep 3, 2024
25a2a77
remove dozer from file and function name
alvrs Sep 3, 2024
6cb9b3f
rename fetchRecordsSql to fetchRecords
alvrs Sep 3, 2024
f2fbaad
use dozer base url as input to fetchRecords
alvrs Sep 3, 2024
c92258e
move error handling into getSnapshot
alvrs Sep 3, 2024
494b0bb
refactors
alvrs Sep 3, 2024
11c5cdd
stylistic changes
alvrs Sep 3, 2024
9adc159
remove unrelated change
alvrs Sep 3, 2024
403179f
review fixes
alvrs Sep 5, 2024
6488f2c
fix export conflict
alvrs Sep 5, 2024
c71eff1
Merge branch 'main' into alvrs/dozer-query
alvrs Sep 5, 2024
1c77240
docs(filter-sync and dozer): explain that there are two sync types 🚗 …
qbzzt Sep 17, 2024
b819a2c
docs(dozer): add metadata query 🚗 (#3186)
qbzzt Sep 17, 2024
a005e3d
Create soft-boats-protect.md
alvrs Sep 18, 2024
805f433
self-review
alvrs Sep 18, 2024
ed21c2f
remove working title
alvrs Sep 18, 2024
e681e95
Merge branch 'main' into alvrs/dozer-query
alvrs Sep 18, 2024
b04a4d8
Merge branch 'main' into alvrs/dozer-query
alvrs Sep 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
rename things
  • Loading branch information
alvrs committed Aug 15, 2024
commit b36d6fe87674395d11514d6d82f0e8c0d97cfca9
6 changes: 3 additions & 3 deletions packages/store-sync/src/dozer/common.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { Table } from "@latticexyz/config";
import { Hex } from "viem";

export type DozerTableQuery = {
export type TableQuery = {
table: Table;
/**
* SQL to filter the records of this table.
Expand All @@ -12,7 +12,7 @@ export type DozerTableQuery = {
sql: string;
};

export type DozerLogFilter = {
export type LogFilter = {
/**
* Filter logs by the table ID.
*/
Expand All @@ -27,4 +27,4 @@ export type DozerLogFilter = {
key1?: Hex;
};

export type DozerSyncFilter = DozerTableQuery | DozerLogFilter;
export type SyncFilter = TableQuery | LogFilter;
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
import { describe, expect, it } from "vitest";
import { fetchRecordsDozerSql } from "./fetchRecordsDozerSql";
import { fetchRecordsSql } from "./fetchRecordsSql";
import mudConfig from "@latticexyz/world/mud.config";
import { selectFrom } from "./selectFrom";

describe("fetch dozer sql", () => {
describe("fetchRecordsSql", () => {
// TODO: set up CI test case for this (requires setting up dozer in CI)
it.skip("should fetch dozer sql", async () => {
const result = await fetchRecordsDozerSql({
const result = await fetchRecordsSql({
dozerUrl: "https://redstone2.dozer.skystrife.xyz/q",
storeAddress: "0x9d05cc196c87104a7196fcca41280729b505dbbf",
queries: [
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { DecodeDozerRecordsResult, DozerQueryResult, decodeDozerRecords } from "./decodeDozerRecords";
import { Hex } from "viem";
import { DozerTableQuery } from "./common";
import { TableQuery } from "./common";
import { Table } from "@latticexyz/config";

type DozerResponseSuccess = {
Expand All @@ -12,13 +12,17 @@ type DozerResponseFail = { msg: string };

type DozerResponse = DozerResponseSuccess | DozerResponseFail;

type FetchDozerSqlArgs = {
function isDozerResponseFail(response: DozerResponse): response is DozerResponseFail {
return "msg" in response;
}

type FetchRecordsSqlArgs = {
dozerUrl: string;
storeAddress: Hex;
queries: DozerTableQuery[];
queries: TableQuery[];
};

type FetchDozerSqlResult =
type FetchRecordsSqlResult =
| {
blockHeight: bigint;
result: {
Expand All @@ -28,15 +32,11 @@ type FetchDozerSqlResult =
}
| undefined;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not blocking but fwiw I found it quite nice to model this sort of thing with arktype, then can parse/validate/return strongly typed response from a JSON API request


function isDozerResponseFail(response: DozerResponse): response is DozerResponseFail {
return "msg" in response;
}

export async function fetchRecordsDozerSql({
export async function fetchRecordsSql({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about just fetchRecords? or query? it's dozer so I think SQL can be implied

dozerUrl,
queries,
storeAddress,
}: FetchDozerSqlArgs): Promise<FetchDozerSqlResult> {
}: FetchRecordsSqlArgs): Promise<FetchRecordsSqlResult> {
const response: DozerResponse = await (
await fetch(dozerUrl, {
Copy link
Member

@holic holic Aug 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since this URL we're fetching is tied to the shape of the request/response, I wonder if this arg should be the endpoint/hostname and we append the path like /q?

method: "POST",
Expand All @@ -57,7 +57,7 @@ export async function fetchRecordsDozerSql({
return;
Copy link
Member

@holic holic Aug 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find it a little weird that a function to fetch a thing returns undefined on a failure rather than throwing or something.

I would usually reserve this behavior for the place where "failure is optional" e.g. getSnapshot or syncToStash

no changes expected, just curious to hear your reasoning

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no i think you're right, can't remember why i put it in here. moved it into getSnapshot now.

}

const result: FetchDozerSqlResult = {
const result: FetchRecordsSqlResult = {
blockHeight: BigInt(response.block_height),
result: response.result.map((records, index) => ({
table: queries[index].table,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,30 +1,30 @@
import { DozerLogFilter, DozerSyncFilter, DozerTableQuery } from "./common";
import { LogFilter, SyncFilter, TableQuery } from "./common";
import { Hex } from "viem";
import { StorageAdapterBlock, SyncFilter } from "../common";
import { fetchRecordsDozerSql } from "./fetchRecordsDozerSql";
import { StorageAdapterBlock, SyncFilter as LegacyLogFilter } from "../common";
import { fetchRecordsSql } from "./fetchRecordsSql";
import { recordToLog } from "../recordToLog";
import { getSnapshot } from "../getSnapshot";
import { getSnapshot as getSnapshotLogs } from "../getSnapshot";
import { bigIntMin, isDefined } from "@latticexyz/common/utils";

export type FetchInitialBlockLogsDozerArgs = {
export type GetSnapshotArgs = {
dozerUrl: string;
storeAddress: Hex;
filters?: DozerSyncFilter[];
filters?: SyncFilter[];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it better or worse to split this into two args: filters and queries?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kinda like the fact that table is shared by both so you can start simple by providing just the table (if you wanna fetch the entire table) and only add sql to the ones where it's needed later

startBlock?: bigint;
chainId: number;
};

export type FetchInitialBlockLogsDozerResult = {
export type GetSnapshotResult = {
initialBlockLogs: StorageAdapterBlock;
};

export async function fetchInitialBlockLogsDozer({
export async function getSnapshot({
dozerUrl,
storeAddress,
filters,
startBlock = 0n,
chainId,
}: FetchInitialBlockLogsDozerArgs): Promise<FetchInitialBlockLogsDozerResult> {
}: GetSnapshotArgs): Promise<GetSnapshotResult> {
const initialBlockLogs: StorageAdapterBlock = { blockNumber: startBlock, logs: [] };

// We execute the list of provided SQL queries for hydration. For performance
Expand All @@ -36,13 +36,13 @@ export async function fetchInitialBlockLogsDozer({
// partial updates), so we only notify consumers of state updates after the
// initial hydration is complete.

const sqlFilters = filters ? (filters.filter((filter) => "sql" in filter) as DozerTableQuery[]) : [];
const sqlFilters = filters ? (filters.filter((filter) => "sql" in filter) as TableQuery[]) : [];

// Execute individual SQL queries as separate requests to parallelize on the backend.
// Each individual request is expected to be executed against the same db state so it
// can't be parallelized.
const dozerTables = (
await Promise.all(sqlFilters.map((filter) => fetchRecordsDozerSql({ dozerUrl, storeAddress, queries: [filter] })))
await Promise.all(sqlFilters.map((filter) => fetchRecordsSql({ dozerUrl, storeAddress, queries: [filter] })))
).filter(isDefined);

if (dozerTables.length > 0) {
Expand All @@ -54,30 +54,30 @@ export async function fetchInitialBlockLogsDozer({
}

// Fetch the tables without SQL filter from the snapshot logs API for better performance.
const snapshotFilters =
const logsFilters =
filters &&
filters
.filter((filter) => !("sql" in filter))
.map((filter) => {
const { table, key0, key1 } = filter as DozerLogFilter;
return { tableId: table.tableId, key0, key1 } as SyncFilter;
const { table, key0, key1 } = filter as LogFilter;
return { tableId: table.tableId, key0, key1 } as LegacyLogFilter;
});

const snapshot =
const logs =
// If no filters are provided, the entire state is fetched
!snapshotFilters || snapshotFilters.length > 0
? await getSnapshot({
!logsFilters || logsFilters.length > 0
? await getSnapshotLogs({
chainId,
address: storeAddress,
filters: snapshotFilters,
filters: logsFilters,
indexerUrl: dozerUrl,
})
: undefined;

// The block number passed in the overall result will be the min of all queries and the snapshot.
if (snapshot) {
initialBlockLogs.blockNumber = bigIntMin(initialBlockLogs.blockNumber, snapshot.blockNumber);
initialBlockLogs.logs = [...initialBlockLogs.logs, ...snapshot.logs];
// The block number passed in the overall result will be the min of all queries and the logs.
if (logs) {
initialBlockLogs.blockNumber = bigIntMin(initialBlockLogs.blockNumber, logs.blockNumber);
initialBlockLogs.logs = [...initialBlockLogs.logs, ...logs.logs];
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ooc what happens if fetchLogs fails but fetchSql doesn't? or vice versa? or some of fetchSql does? do we end up in a weird state?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah i think before that was the case, now all of getSnapshot fails if one of the individual requests fails

return { initialBlockLogs };
Expand Down
4 changes: 2 additions & 2 deletions packages/store-sync/src/dozer/index.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
export * from "./common";
export * from "./fetchRecordsDozerSql";
export * from "./fetchRecordsSql";
export * from "./selectFrom";
export * from "./fetchInitialBlockLogsDozer";
export * from "./getSnapshot";
4 changes: 2 additions & 2 deletions packages/store-sync/src/dozer/selectFrom.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
import { Table } from "@latticexyz/config";
import { DozerTableQuery } from "./common";
import { TableQuery } from "./common";

// For autocompletion but still allowing all SQL strings
export type Where<table extends Table> = `"${keyof table["schema"] & string}"` | (string & {});
Copy link
Member

@holic holic Aug 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


export type SelectFromArgs<table extends Table> = { table: table; where?: Where<table>; limit?: number };

export function selectFrom<table extends Table>({ table, where, limit }: SelectFromArgs<table>): DozerTableQuery {
export function selectFrom<table extends Table>({ table, where, limit }: SelectFromArgs<table>): TableQuery {
const dozerTableLabel = table.namespace === "" ? table.name : `${table.namespace}__${table.name}`;
return {
table: table,
Expand Down