Bad block pruning?

[xlc]

One of our RPC node is missing data for a block and only that block (AFAIK)
Endpoint: wss://karura-rpc-0.aca-api.network
Block number: 3467086
Block hash: 0xd13a656c8c4cd7a6f7d03db8209eee9c597edffba1b7ee2dc40844089e10b21a

I tried query +- 5 blocks and they all good.

The RPC node is running acala/karura-node:2.12.0 which uses 0.9.33 Substrate

This is the CLI args

    "--base-path=/acala/data", \
    "--chain=karura", \
    "--pruning=archive", \
    "--ws-external", \
    "--rpc-external", \
    "--rpc-cors=all", \
    "--ws-port=9944", \
    "--rpc-port=9933", \
    "--ws-max-connections=1200", \
    "--in-peers=100", \
    "--listen-addr=/ip4/0.0.0.0/tcp/30333", \
    "--listen-addr=/ip4/0.0.0.0/tcp/30334/ws", \
    "--state-cache-size=0", \
    "--db-cache=2048", \
    "--max-runtime-instances=16", \
    "--", \
    "--chain=kusama", \
    "--listen-addr=/ip4/0.0.0.0/tcp/30335", \
    "--listen-addr=/ip4/0.0.0.0/tcp/30336/ws" \

[xlc]

Also in our Mandala testnet, without --keep-blocks=archive, only the block body of the last block is preserved. Not 100% sure if it is related bug that could indicate a bug somewhere. We don't have this issue for Karura & Acala so we didn't report it before as it could just be our setup wasn't exactly right.

[shunjizhan]

Just wanna add that for this particular node, it's block 3467086 that has the issue. However, it might not be deterministic, we have seen similar issues at different blocks for other karura nodes.

[bkchr]

Also in our Mandala testnet, without --keep-blocks=archive, only the block body of the last block is preserved.

How do you verify this? Also just calling getBlock?

[bkchr]

One of our RPC node is missing data for a block and only that block (AFAIK)

This block is finalized?

[xlc]

How do you verify this? Also just calling getBlock?

Yes get block gives empty data

This block is finalized?

Yes

[bkchr]

Have you seen this on multiple nodes? If not, anything special that happened with this node? Did you changed the block pruning settings at some point?

[xlc]

This is happening on multiple nodes on different block height including our own node and community nodes. We did not change the block pruning settings.

[bkchr]

@arkpar do you have any idea?

[arkpar]

Does it get exported with export-blocks command?

[xlc]

acala export-blocks --base-path /acala/data --chain=karura --from 3467086 --to 3467089
/acala/data --chain=karura --from 3467086 --to 3467089
2023-02-07 09:09:09 DB path: /acala/data/chains/karura/db/full
2023-02-07 09:09:09 Exporting blocks from #3467086 to #3467089
2023-02-07 09:09:09 Essential task `basic-block-import-worker` failed. Shutting down service.
2023-02-07 09:09:09 Essential task `transaction-pool-task-0` failed. Shutting down service.
2023-02-07 09:09:09 Essential task `transaction-pool-task-1` failed. Shutting down service.
2023-02-07 09:09:09 Essential task `txpool-background` failed. Shutting down service.

Tried other block range and able to export the blocks.

[xlc]

I will guess the only way to restore those missing blocks is resync?