feat: make historical epochs state #362

golddydev · 2025-11-16T19:40:57Z

Description

This PR introduces a new dedicated historical-epochs-state module for managing historical epoch data with persistent storage, refactors the existing epochs-state module to remove historical tracking responsibilities, and improves CBOR encoding for EpochActivityMessage with custom u128 encoder/decoder.

Changes

1. New historical-epochs-state Module

Purpose: Dedicated module for storing and querying historical epoch activity data for Blockfrost API compatibility
Architecture: Volatile + Immutable dual-layer storage
- Volatile: In-memory state for recent epochs (rollback support)
- Immutable: Persistent disk storage using fjall LSM-tree database
Features:
- Automatic pruning based on security parameter k
- Background persistence worker (bounded channel to prevent backpressure)
- Bounded pending queue (5 max) to prevent memory exhaustion
- Rollback support for chain reorganizations
- Query handlers for: GetEpochInfo, GetNextEpochs, GetPreviousEpochs

2. REST API Improvements

Fixed: handle_epoch_info_blockfrost now returns latest epoch when numeric param equals latest
Enhanced: Better separation between current epoch queries (epochs-state) vs historical queries

Fixes

1. Epoch Activity Message Timing

Fixed: Publish epoch activity message before evolving nonce
Impact: Ensures consumers receive correct epoch activity message (with corresponding epoch's nonce)
2. Byron Block Mint Handling
Fixed: Handle mint for Byron-era blocks correctly

Related Issue(s)

Related to #364

How was this tested?

Added test cases for historical_epochs_state to test volatile's prune, correct persistent and reading from disk.
Added test cases for minicbor encode/decode for u128

Checklist

My code builds and passes local tests
I added/updated tests for my changes, where applicable
I added some comments
CI is green for this PR

Impact / Side effects

This PR off-loads the memory usage which is consumed by epochs_state to serve in-memory epochs history.

Reviewer notes / Areas to focus

historical_epochs_state module is the main changes of this PR.

…t latest but number

… to common serde crate

Copilot

Pull Request Overview

This PR introduces a new historical_epochs_state module to track historical epoch information separately from the main epochs_state module, improving separation of concerns and enabling optional historical data storage. The refactoring moves epoch history storage from an in-memory DashMap to a persistent fjall database with proper volatile/immutable state separation.

Key Changes

New historical_epochs_state module with persistent storage using fjall database
Refactored epochs_state to remove history tracking and delegate to the new module
Updated REST handlers to query historical epochs from the new dedicated state module
Added CBOR serialization support for EpochActivityMessage and related types

Reviewed Changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
`modules/historical_epochs_state/*`	New module implementing persistent historical epoch storage with volatile/immutable state separation
`modules/epochs_state/src/epochs_state.rs`	Removed epoch history logic and simplified to focus on current epoch tracking
`modules/epochs_state/src/state.rs`	Changed `handle_mint` to accept `Option<&[u8]>` for issuer_vkey
`modules/epochs_state/src/epochs_history.rs`	Deleted - functionality moved to new module
`modules/epochs_state/src/store_config.rs`	Deleted - no longer needed
`modules/rest_blockfrost/src/handlers/epochs.rs`	Updated to route historical queries to new module and validate epoch bounds
`modules/rest_blockfrost/src/handlers_config.rs`	Added historical epochs query topic configuration
`common/src/queries/epochs.rs`	Added `DEFAULT_HISTORICAL_EPOCHS_QUERY_TOPIC` constant
`common/src/messages.rs`	Added minicbor serialization derives to `EpochActivityMessage`
`common/src/protocol_params.rs`	Added minicbor serialization to `Nonce` types
`common/src/cbor.rs`	New module with custom u128 CBOR codec
`processes/omnibus/*`	Registered new module and added configuration section
`modules/accounts_state/src/accounts_state.rs`	Fixed log message from "subscriber" to "publisher"

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

common/src/cbor.rs

modules/rest_blockfrost/src/handlers/epochs.rs

modules/historical_epochs_state/src/historical_epochs_state.rs

modules/rest_blockfrost/src/handlers/epochs.rs

modules/historical_epochs_state/src/immutable_historical_epochs_state.rs

modules/epochs_state/src/epochs_state.rs

…ror traces

SupernaviX · 2025-11-17T20:02:22Z

modules/historical_epochs_state/src/immutable_historical_epochs_state.rs

+        for epoch in range {
+            let slice = self.epochs_history.get(Self::make_epoch_key(epoch))?;


The fjall library has a range method which takes a range of keys and fetches everything in that range at once. Probably faster to use that and let fjall batch stuff.

Fixed
6a951d1

SupernaviX · 2025-11-17T20:06:17Z

modules/historical_epochs_state/src/state.rs

+        let db_path = if Path::new(&config.db_path).is_relative() {
+            PathBuf::from(env!("CARGO_MANIFEST_DIR")).join(&config.db_path)
+        } else {
+            PathBuf::from(&config.db_path)
+        };


CARGO_MANIFEST_DIR is set at compile time, to a path on the compiling machine. It won't generally work to use it at runtime.

Honestly, we probably should just pass relative paths along unchanged. Filesystem APIs will treat them as relative to the PWD at runtime, which is probably the behavior we want for them.

28d9e0b

Thank you for pointing this out.
Now it uses relative path (both accounts and epochs historical state) and prefix default db path with fjall-* to add them .gitignore easily.

…ath with fjall and update gitignore files to ignore all db related files

whankinsiv

Great job! Could you please explain why this modules ready_to_prune logic is offset by one epoch compared to historical_accounts_state.

whankinsiv · 2025-11-18T01:41:52Z

modules/historical_epochs_state/src/immutable_historical_epochs_state.rs

+    pub fn new(path: impl AsRef<Path>) -> Result<Self> {
+        let cfg = fjall::Config::new(path)
+            // 4MB write buffer since EpochActivityMessage is not that big


Could you please add a clear-on-start config option.

modules/historical_epochs_state/src/state.rs

lowhung

Nice to see the historical responsbility split out like this!

golddydev added 7 commits November 16, 2025 19:52

feat: implement historical epochs state

abe0e9f

fix: handle mint for byron blocks

acd725c

fix: publish epoch activity message before evolving nonce

276ade5

fix: epoch info endpoint to return latest epoch even when param is no…

f3df74d

…t latest but number

fix: cargo shear

9403ddd

fix: space indent

99c8a1e

fix: typo

8514496

github-actions bot mentioned this pull request Nov 17, 2025

📊 Weekly Status - Week of 2025-11-17 #363

Open

tests: add test cases for historical epochs state and add rc features…

91f5f1f

… to common serde crate

golddydev requested a review from Copilot November 17, 2025 14:23

Copilot started reviewing on behalf of golddydev November 17, 2025 14:24 View session

Copilot finished reviewing on behalf of golddydev November 17, 2025 14:25

Copilot AI reviewed Nov 17, 2025

View reviewed changes

golddydev added 2 commits November 17, 2025 16:05

fix: typo, add max pending to historical epochs state pending, add er…

a5f74c9

…ror traces

refactor: u128 cbor encode and add test cases

ae44ba6

golddydev marked this pull request as ready for review November 17, 2025 15:32

golddydev requested review from SupernaviX, lowhung and whankinsiv November 17, 2025 15:35

SupernaviX reviewed Nov 17, 2025

View reviewed changes

golddydev added 2 commits November 18, 2025 06:36

refactor: use fjall's range instead of manual get several times

6a951d1

refactor: use relative path for historical storage and prefix their p…

28d9e0b

…ath with fjall and update gitignore files to ignore all db related files

golddydev requested a review from SupernaviX November 18, 2025 06:05

Merge branch 'main' into gd/historical-epochs-state

7b5fe6b

SupernaviX approved these changes Nov 18, 2025

View reviewed changes

whankinsiv reviewed Nov 18, 2025

View reviewed changes

lowhung approved these changes Nov 18, 2025

View reviewed changes

		for epoch in range {
		let slice = self.epochs_history.get(Self::make_epoch_key(epoch))?;

feat: make historical epochs state #362

Are you sure you want to change the base?

feat: make historical epochs state #362

Uh oh!

Conversation

golddydev commented Nov 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Fixes

Related Issue(s)

How was this tested?

Checklist

Impact / Side effects

Reviewer notes / Areas to focus

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Key Changes

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SupernaviX Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

golddydev Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

SupernaviX Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

golddydev Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

whankinsiv left a comment

Choose a reason for hiding this comment

Uh oh!

whankinsiv Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lowhung left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

golddydev commented Nov 16, 2025 •

edited

Loading