enhancement: forc index status: report errors and indexed blocks

[lostman]

Report indexed blocks

The only way to observe that an indexer is working is to look at the data in the database.

Or, what I do practically all the time, add this to an indexer:

info!("Block #{}", block_data.height);

It works fine, but every time I switch a branch, I must stash/apply this. 😢

We need a more convenient way to check that "it's working."

We can do that by making forc index status gather information about indexed blocks for each indexer (MAX(block_height) on the indexmetadataentity table) and display it.

Save and display status/errors

If an indexer stops for any reason, we should save the error and display it in the output of forc index status.

This may include WASM error (and with WASM error codes PR, we'll get more useful errors: #1293) or perhaps some database error that causes the indexer to miss blocks.

This PR #1297 introduces a trigger on indexmetadataentity to ensure no blocks are missing (more info here: #1150). It will stop the indexer from progressing. However, the only way to check what happened is to look through the logs:

Aug 31 10:50:10.335 ERROR fuel_indexer::database: Failed to put_object: Database(PgDatabaseError { severity: Error, code: "P0001", message: "fuel_indexer_test_index1.indexmetadataentity attempted to insert value with block_height = 5 while last processed block_height = 3. block_height values must be consecutive.", detail: None, hint: None, position: None, where: Some("PL/pgSQL function ensure_block_height_consecutive() line 8 at RAISE"), schema: None, table: None, column: None, data_type: None, constraint: None, file: Some("pl_exec.c"), line: Some(3907), routine: Some("exec_stmt_raise") })

We can save this (and other) error messages in the database and include them in forc index status.

Example:

─ fuellabs
   └─ explorer
      • id: 78
      • created_at: 2023-08-28 10:13:28.617545 UTC
      • pubkey: None
      • indexed blocks: 12345
      • status: stopped with error: fuellabs_explorer.indexmetadataentity: attempted to insert value with block_height = 5 while last processed block_height = 3. block_height values must be consecutive.

Or simply:

      • status: active

If everything is well.

[ra0x3]

Some UX nits from me on this:

• Maybe "active" or "error" status could be colored red/green depending on the status
• If the indexer has an end_block maybe we could show progress (as a percent) vs that end block
• Remove the id field from the response (that's internal to the DB and not of use to the user)