Description
Description
As a private blockchain with real time transaction requirements with high performance, the system should handle a high tps at a low block time so that we can receive transaction responses in real time when any transaction is done. We have preformed a load test on Hyperledger Besu using IBFT 2.0 sending a continuous load of 200 tps sent across multiple nodes. The script that sent the transactions was handling the nonce value in a thread safe manner to be able to send transactions concurrently. However due to the continuous load of transactions some transactions were sent with an invalid nonce (either lower or higher nonce value than the sender account nonce). The transactions with invalid nonce remained in the transaction pool even after the load stopped, and were never removed after that. These transactions filled up the transaction pool and were being processed on every block creation making transaction processing slower.
Acceptance Criteria
- Transactions with an invalid nonce should be rejected and not added to the transaction pool.
Steps to Reproduce (Bug)
Steps done to reproduce the issue (we were able to reproduce it after starting a fresh installation), a script was created to do the following steps:
- Besu setup we had were 7 nodes with a block time of 2 seconds and tx-pool-retention-hours=1
- Created 400 accounts and filled them up with 1 Eth
- A loop that ran every 2 seconds (block time) sent concurrently transactions from 1 account to the next account sending 0.1 Eth so as to simulate 400 transaction sent for 1 block, after every loop the transactions were flipped sending back the amount, to keep transactions running.
- After some time of the script running logs can show pending transactions increasing (which had invalid nonce), reaching a point were these transactions are stuck in the systems with logs showing them being processed every time.
Expected behavior: Transactions with invalid nonce should be rejected and not stored in the transaction pool
Actual behavior: Transaction with invalid nonce remained in the transaction pool even after the retention hour has passed (-tx-pool-retention-hours=1)
Frequency: Can be reproduced every time
Versions (Add all that apply)
- Software version: besu/v20.10.3/linux-x86_64/openjdk-java-11
- Java version: openjdk 11.0.9.1 2020-11-04 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.9.1+1-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.9.1+1-LTS, mixed mode, sharing) - OS Name & Version: CentOS Linux release 7.9.2009 (Core)
- Kernel Version: Linux ldp-node2 3.10.0-1160.11.1.el7.x86_64 Update Jenkins Build for Pantheon → Besu considerations #1 SMP Fri Dec 18 16:34:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
- Cloud VM, type, size: Local VM 4 core, 4 GB RAM