feat: (experimental) optional fee estimator with different algorithms #4477

yangby-cryptape · 2024-06-04T06:55:32Z

What problem does this PR solve?

Warning

This feature is experimental.
I have done lots of manual tests before submit this PR, but more production environment tests are still required.

Add an optional fee estimator with different algorithms and a RPC method for it.

Description

I do lots manual tests for several day, I don't sure which algorithm is better, so I just submit both of them.

Let's make choice after collect more data in production environment (both mainnet and testnet).
Only 4 levels of priorities for the fee estimator.

Different algorithms have different APIs, I just use some constants as default parameters to avoid specialized parameters, to make the public API more generic.

Current, there are 2 built-in algorithms:

Confirmation Fraction

Migrated from this CKB PR.

👉🏼 Sometimes, it returns unreasonable result, I haven't figured it out yet, since I couldn't reproduce it manually.
Weight-Units Flow

A variant of the Weight-Units Flow Fee Estimator for Bitcoin.

The main difference is I use the block interval to replace the time interval, to avoid unit conversion with random distributions, since our APIs require blocks as the target unit for estimation.

The details of this algorithm are written in the head section of that source code file.

Merged fallback fee estimates algorithm from #4465.

Usage

Append the following content to ckb.toml:

[fee_estimator]
# Specifies the fee estimates algorithm. Current algorithms: ConfirmationFraction, WeightUnitsFlow.
algorithm = "WeightUnitsFlow"

And enable JSON-RPC API module Experiment.

After the CKB node started, try the following command:

echo '{ "id": 1, "jsonrpc": "2.0", "method": "estimate_fee_rate", "params": [] }' \
    | curl -s -H "Content-Type: application/json" -d @- "http://localhost:8114" \
    | jq

An example of result is:

{
  "jsonrpc": "2.0",
  "result":"0x3e8",
  "id": 1
}

What is changed and how it works?

What's Changed:

Related changes

PR to update owner/repo:
Need to cherry-pick to the release branch

Check List

Tests

Unit test
Integration test

Release note

Title Only: Include only the PR title in the release note.

eval-exec · 2024-06-07T07:08:51Z

util/fee-estimator/src/estimator/confirmation_fraction.rs

+impl Algorithm {
+    pub fn update_ibd_state(&mut self, in_ibd: bool) {
+        if self.is_ready {
+            if in_ibd {
+                self.clear();
+                self.is_ready = false;
+            }
+        } else if !in_ibd {
+            self.clear();
+            self.is_ready = true;
+        }
+    }


update_ibd_state is executed every time ckb-chain finishes processing a block, is this the expected behavior?

Or should update_ibd_state only be executed when the ibd_state of ckb changes from true to false?

If I was correct, the IBD state could change to true from false, when a CKB node lost Internet connections for a long time.

I believe that when CKB leaves IBD mode, self.ibd_finished is set to true only once. If CKB loses internet connection after 24 hours, self.ibd_finished will always remain true.

Cc. @driftluo @zhangsoledad

/// Return whether chain is in initial block download pub fn is_initial_block_download(&self) -> bool { // Once this function has returned false, it must remain false. if self.ibd_finished.load(Ordering::Acquire) { false } else if unix_time_as_millis().saturating_sub(self.snapshot().tip_header().timestamp()) > MAX_TIP_AGE { true } else { self.ibd_finished.store(true, Ordering::Release); false } }

util/fee-estimator/src/estimator/weight_units_flow.rs

…ture/integrated-fee-estimator

eval-exec · 2024-09-22T14:25:43Z

util/fee-estimator/src/constants.rs

+use ckb_types::core::{BlockNumber, FeeRate};
+
+/// Average block interval.
+pub(crate) const AVG_BLOCK_INTERVAL: u64 = (MAX_BLOCK_INTERVAL + MIN_BLOCK_INTERVAL) / 2;


The AVG_BLOCK_INTERVAL is calculated as (48 + 8) / 2, which equals 28 seconds. However, according to https://explorer.nervos.org/charts/block-time-distribution, most block intervals are less than 10 seconds.

eval-exec · 2024-09-22T14:28:06Z

util/fee-estimator/src/constants.rs

+pub(crate) const MIN_TARGET: BlockNumber = TX_PROPOSAL_WINDOW.closest() + 1;
+
+/// Lowest fee rate.
+pub(crate) const LOWEST_FEE_RATE: FeeRate = FeeRate::from_u64(1000);


Should we reuse DEFAULT_MIN_FEE_RATE from ckb-app-config?

eval-exec · 2024-09-22T14:29:53Z

util/fee-estimator/src/error.rs

+    #[error("dummy fee estimator is used")]
+    Dummy,
+    /// Not ready for do estimate.
+    #[error("not ready")]


How about giving a more verbose errro: #[error("not ready, node is in IBD mode")]

eval-exec · 2024-09-22T14:52:41Z

util/fee-estimator/src/estimator/confirmation_fraction.rs

+    /// confirm_blocks => bucket index => confirmed txs count
+    confirm_blocks_to_confirmed_txs: Vec<Vec<f64>>,
+    /// confirm_blocks => bucket index => failed txs count
+    confirm_blocks_to_failed_txs: Vec<Vec<f64>>,


It seems that confirm_blocks_to_failed_txs's length is always 1?
I didn't find which method append/insert Vec<f64> to confirm_blocks_to_failed_txs after initialization.

feat: add estimate_fee_rate rpc

d6b5eb9

yangby-cryptape requested a review from a team as a code owner June 4, 2024 06:55

yangby-cryptape requested review from quake, chenyukang, zhangsoledad and eval-exec and removed request for a team June 4, 2024 06:55

eval-exec reviewed Jun 7, 2024

View reviewed changes

util/fee-estimator/src/estimator/weight_units_flow.rs Outdated Show resolved Hide resolved

yangby-cryptape added 3 commits June 25, 2024 18:38

feat: (experimental) optional fee estimator with different algorithms

d3fa319

Merge remote-tracking branch 'quake:quake/estimate-fee-rate' into fea…

69dbc24

…ture/integrated-fee-estimator

Merge branch 'develop' into feature/integrated-fee-estimator

1172d2d

yangby-cryptape force-pushed the feature/integrated-fee-estimator branch from f7f60e0 to 1172d2d Compare July 4, 2024 06:09

quake mentioned this pull request Jul 19, 2024

feat: add estimate_fee_rate rpc #4465

Closed

eval-exec reviewed Sep 22, 2024

View reviewed changes

eval-exec requested a review from driftluo September 22, 2024 14:40

eval-exec added t:enhancement Type: Feature, refactoring. b:rpc Break RPC interface m:tx-pool labels Sep 22, 2024

eval-exec reviewed Sep 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: (experimental) optional fee estimator with different algorithms #4477

feat: (experimental) optional fee estimator with different algorithms #4477

yangby-cryptape commented Jun 4, 2024 •

edited

Loading

eval-exec Jun 7, 2024

yangby-cryptape Jul 4, 2024

eval-exec Sep 22, 2024 •

edited

Loading

eval-exec Sep 22, 2024

eval-exec Sep 22, 2024

eval-exec Sep 22, 2024 •

edited

Loading

eval-exec Sep 22, 2024 •

edited

Loading

feat: (experimental) optional fee estimator with different algorithms #4477

Are you sure you want to change the base?

feat: (experimental) optional fee estimator with different algorithms #4477

Conversation

yangby-cryptape commented Jun 4, 2024 • edited Loading

What problem does this PR solve?

Description

Usage

What is changed and how it works?

Related changes

Check List

Release note

eval-exec Jun 7, 2024

Choose a reason for hiding this comment

yangby-cryptape Jul 4, 2024

Choose a reason for hiding this comment

eval-exec Sep 22, 2024 • edited Loading

Choose a reason for hiding this comment

eval-exec Sep 22, 2024

Choose a reason for hiding this comment

eval-exec Sep 22, 2024

Choose a reason for hiding this comment

eval-exec Sep 22, 2024 • edited Loading

Choose a reason for hiding this comment

eval-exec Sep 22, 2024 • edited Loading

Choose a reason for hiding this comment

yangby-cryptape commented Jun 4, 2024 •

edited

Loading

eval-exec Sep 22, 2024 •

edited

Loading

eval-exec Sep 22, 2024 •

edited

Loading

eval-exec Sep 22, 2024 •

edited

Loading