Purgeable database interfaces

[nytzuga]

Context and scope

The main goal of this feature request is to allow node operator to configure their nodes to keep the last N accepted blocks persistent, purging anything older.

Feature requirements

A new optional configuration setting should be added, to specify the last N blocks to be kept in the database after a cleanup. If this setting is not provided, it is assumed the full state should be kept in the database. Such configuration should also be provided in time terms (1 year or interpreted as the last N blocks if their value is just an integer number).

Technical requirements

The database interface should be updated to introduce a new function to delete the last N elements. It is an implementation detail of how the deletion happens and how the spaces are reclaimed. For instance, under a few circumstances, it would be even faster to a new database, copy the last N elements, close the old database, and delete their files at the OS level.

The avalanchego should call such a function on demand. Each subnet is not aware of this setting nor they care. By doing so, all subnets inherit this bevahiour effortlessly. This configuration should be transversal to all subnets. Maybe there could be an option for some future subnet that may need to be a full node in order to operate. That is outside of the scope of this documentation.

sequenceDiagram
	AvalancheGo->>+SubnetDB: Purge, keep last 1000 blocks
	SubnetDB  ->>+  AvalancheGo: Queued
	SubnetDB  ->>+  InnerDB: Delete everything but the last 1000 blocks
	AvalancheGo  ->>+  SubnetDB: Add new block
	AvalancheGo  ->>+  SubnetDB: Add new block
	AvalancheGo  ->>+  SubnetDB: Add new block
	InnerDB  ->>+  SubnetDB: Done
	AvalancheGo  ->>+  SubnetDB: Add new block
	AvalancheGo  ->>+  SubnetDB: Add new block

Loading

When the purge function is being called, with the number of last blocks to preserve, each database should delete old data as efficiently as possible, ideally without hijacking the entire database. Ideally, a generic implementation should be included, probably creating a new copy of the database, copying only the last N blocks, and deleting the old database when the copy is finished. Again, that implementation is outside of the requirement of this documentation.

Probably a realistic implementation would look like this:

sequenceDiagram
    AvalancheGo->>+SubnetDB: Purge, keep last 1000 blocks
    SubnetDB ->>+ AvalancheGo: Queued
    alt Start migration
        SubnetDB ->>+ InnerDB: Read the last 1000 blocks
        InnerDB ->>+ SubnetDB: cursor 1000 records
        SubnetDB ->>+ NewInnerDB: write 1000 records
    end
    alt New blocks are being added
        AvalancheGo ->>+ SubnetDB: Add new block
        SubnetDB ->>+ InnerDB: Add new block
        SubnetDB ->>+ NewInnerDB: Add new block
        AvalancheGo ->>+ SubnetDB: Add new block
        SubnetDB ->>+ InnerDB: Add new block
        SubnetDB ->>+ NewInnerDB: Add new block
    end
    InnerDB ->>+ SubnetDB: Done purging is done
    SubnetDB ->>+ InnerDB: Delete and close
    SubnetDB ->>+ NewInnerDB: You are promoted
    AvalancheGo ->>+ SubnetDB: Add new block
    SubnetDB ->>+ NewInnerDB: Add new block
    AvalancheGo ->>+ SubnetDB: Add new block
    SubnetDB ->>+ NewInnerDB: Add new block

Loading

Discussion and alternatives

A generic solution could be implemented at a higher level, but I firmly believe that having an interface with a default implementation is better. Having an interface will allow specific optimizations for flavour of supported databases.

Open questions

What are the necessary changes in the interface to be future-proof for other subnets?

[github-actions]

This issue has become stale because it has been open 60 days with no activity. Adding the lifecycle/frozen label will cause this issue to ignore lifecycle events.

[joshua-kim]

Related: #1701