[FEATURE REQUEST] Set DEFAULT_LOWEST_INVALID_KNOWN_NONCE_CACHE configurable, like others transaction pool options

[tzdesing]

Description

We are In a private free gas chain , using QBFT protocol, to implement a national CBDC, when a trasanction get stucked in TX_POOL, that transactions are included in cache. We are unable to send a new transaction with higher gas, so the only way is set our TX_POOL_MAX_SIZE to 0 , and back to default value, then we are able to sent new transactons, but , the invalid transaction was included in cache of validator and we are unable to reset then. because of network rules. So always we sent new transactions BESU will check this cache, and invalidate again because we have a higher nonce.

Acceptance Criteria

IF we are able to set LOWEST_INVALID_KNOWN_NONCE_CACHE = 0 on bootstrap of validators nodes, we can transact normally and transactions will not being invalidated every time, because we haven't a cache, in our case this cache isn't necessary.

[alexcostars]

We are facing the same problem. In our case, we have a lot of accounts that as blocked (all transactions signed by these accounts are getting stuck in TX_POOL).

One other option is a command line option that disables invalid nonce cache control.

[matthew1001]

I raised a discussion on discord about exactly this issue a couple of days ago: https://discord.com/channels/905194001349627914/905205502940696607/1164623046389272699

I was considering a PR that would prevent invalid TXs entering the invalid nonce cache if the reason was NONCE_TOO_LOW (which is the reason I've seen it happening with). The NONCE_TOO_LOW error seems to be the least sensible reason to set the invalid nonce cache to that value - because by definition, you couldn't submit a new TX at that nonce anyway.

I'll potentially raise that PR, but I think the new layered pool implementation doesn't use an invalid nonce cache. So you could set --tx-pool=layered and see if that prevents the invalid nonce cache from blocking up the pool.

[matthew1001]

I've raised #6067 which I think could help in a large number of the cases.

[matthew1001]

Note that I think this issue which I raised yesterday is closely related to the scenario described:

#6043

Currently even if you set --tx-pool-price-bump=0 you cannot replace existing transactions. I think for QBFT, permissioned deployments it is very difficult to manage the pool through set its max size, or its retention time, when problematic transactions make their way into the pool.

[alexcostars]

@matthew1001 as you mentioned, there is a new layered pool implementation. Has this new implementation already been released?

[matthew1001]

@matthew1001 as you mentioned, there is a new layered pool implementation. Has this new implementation already been released?

Yes it's available in 23.10.0 as the default pool implementation.

[matthew1001]

Before 23.10.0 you need to set --Xlayered-tx-pool=true to use it

[fab-10]

the invalid known nonce cache was introduced as a workaround to mitigate the nonce gap problem while we where introducing the new layered txpool, that solves this problem in a much better way. So if you are running Besu version >= 23.4.1 I strongly recommend to use it instead of the legacy one. Since version 23.10.0 it is the default

Said that we see that there are some specific use cases for private networks that are not supported, and there is the opportunity to create a txpool tailored for them.

[alexcostars]

Excellent advice. We will enable this option, but we are using 23.4.1 version and there is no documented option related --Xlayered-tx-pool=true (https://besu.hyperledger.org/23.4.1/public-networks/reference/cli/options and https://besu.hyperledger.org/23.4.1/private-networks/reference/cli/options), then I suggest to include this command line in the official documentation.

There is an equivalent ENV variable to set this config?

[fab-10]

It is not documented in that version, because we are not documenting experimental or hidden features, but you can read about the options, directly in the PR #5290

for the env variable try BESU_XLAYERED_TX_POOL=true

[fab-10]

FYI: This PR is an improvement to allow for easier tx replacement

[alexcostars]

@fab-10,
We recently ran into this issue on one of our addresses (caching invalid nonces).

I understand and agree with your suggestion to enable the parameters (or migrate to 23.10), but we are trying to reproduce the error to document and justify the configuration change, but we are not succeeding. When we didn't want to generate the error it occurred, but now that we want to generate it we can't.

Could you help us understand how we can reproduce the nonce gap scenario in version 23.4? Except network problems and spam transactions (mentioned in the PR), is there any practical way to cause the inclusion of an address in the cache of invalid nonces?

[fab-10]

@alexcostars could you share the configuration and parameters used to start Besu? it will also help to know about your setup, how many nodes and from which nodes you send the txs.