Merge remote-tracking branch 'origin/main' into issue-3810-no-empty-b…

…lock

.github/workflows/container-security-scan.yml

@@ -8,7 +8,7 @@ on:
         required: false
         default: 'develop'      
   schedule:
-    # Start of the hour is the busy time. Scheule it to run 8:17am UTC
+    # Start of the hour is the busy time. Schedule it to run 8:17am UTC
     - cron:  '17 8 * * *'
 
 jobs:

CHANGELOG.md

@@ -1,6 +1,25 @@
 # Changelog
 
-## Next Release
+## Next release
+
+### Breaking Changes
+
+### Additions and Improvements
+- `--Xsnapsync-bft-enabled` option enables experimental support for snap sync with IBFT/QBFT permissioned Bonsai-DB chains [#7140](https://github.com/hyperledger/besu/pull/7140)
+- Add support to load external profiles using `--profile` [#7265](https://github.com/hyperledger/besu/issues/7265)
+- `privacy-nonce-always-increments` option enables private transactions to always increment the nonce, even if the transaction is invalid [#6593](https://github.com/hyperledger/besu/pull/6593)
+- Added `block-test` subcommand to the evmtool which runs blockchain reference tests [#7293](https://github.com/hyperledger/besu/pull/7293)
+
+### Bug fixes
+
+## 24.7.0
+
+### Upcoming Breaking Changes
+- Receipt compaction will be enabled by default in a future version of Besu. After this change it will not be possible to downgrade to the previous Besu version.
+- PKI-backed QBFT will be removed in a future version of Besu. Other forms of QBFT will remain unchanged.
+- --Xbonsai-limit-trie-logs-enabled is deprecated, use --bonsai-limit-trie-logs-enabled instead
+- --Xbonsai-trie-logs-pruning-window-size is deprecated, use --bonsai-trie-logs-pruning-window-size instead
+- `besu storage x-trie-log` subcommand is deprecated, use `besu storage trie-log` instead
 
 ### Breaking Changes
 - `Xp2p-peer-lower-bound` has been removed. [#7247](https://github.com/hyperledger/besu/pull/7247)
@@ -12,13 +31,21 @@
 - A full and up to date implementation of EOF for Prague [#7169](https://github.com/hyperledger/besu/pull/7169)
 - Add Subnet-Based Peer Permissions.  [#7168](https://github.com/hyperledger/besu/pull/7168)
 - Reduce lock contention on transaction pool when building a block [#7180](https://github.com/hyperledger/besu/pull/7180)
+- Update Docker base image to Ubuntu 24.04 [#7251](https://github.com/hyperledger/besu/pull/7251)
+- Add LUKSO as predefined network name [#7223](https://github.com/hyperledger/besu/pull/7223)
+- Refactored how code, initcode, and max stack size are configured in forks. [#7245](https://github.com/hyperledger/besu/pull/7245)
+- Nodes in a permissioned chain maintain (and retry) connections to bootnodes [#7257](https://github.com/hyperledger/besu/pull/7257)
+- Promote experimental `besu storage x-trie-log` subcommand to production-ready [#7278](https://github.com/hyperledger/besu/pull/7278)
+- Enhanced BFT round-change diagnostics [#7271](https://github.com/hyperledger/besu/pull/7271)
 
 ### Bug fixes
 - Validation errors ignored in accounts-allowlist and empty list [#7138](https://github.com/hyperledger/besu/issues/7138)
 - Fix "Invalid block detected" for BFT chains using Bonsai DB [#7204](https://github.com/hyperledger/besu/pull/7204)
 - Fix "Could not confirm best peer had pivot block" [#7109](https://github.com/hyperledger/besu/issues/7109)
 - Fix "Chain Download Halt" [#6884](https://github.com/hyperledger/besu/issues/6884)
 
+
+
 ## 24.6.0
 
 ### Breaking Changes
@@ -2746,7 +2773,7 @@ If you have existing private transactions, see [migration details](docs/Private-
 
 This can be enabled using the `--rpc-http-api TRACE` CLI flag.  There are some philosophical differences between Besu and other implementations that are outlined in [trace_rpc_apis](docs/trace_rpc_apis.md).
 
-- Ability to automatically detect Docker NAT settings from inside the conainter.
+- Ability to automatically detect Docker NAT settings from inside the container.
 
 The default NAT method (AUTO) can detect this so no user intervention is required to enable this.
 
@@ -2969,7 +2996,7 @@ For compatibility with ETC Agharta upgrade, use 1.3.7 or later.
 
 - Update Governance and Code of Conduct verbiage [\#120](https://github.com/hyperledger/besu/pull/120)
 - Fix private transaction root mismatch [\#118](https://github.com/hyperledger/besu/pull/118)
-- Programatically enforce plugin CLI variable names [\#117](https://github.com/hyperledger/besu/pull/117)
+- Programmatically enforce plugin CLI variable names [\#117](https://github.com/hyperledger/besu/pull/117)
 - Additional unit test for selecting replaced pending transactions [\#116](https://github.com/hyperledger/besu/pull/116)
 - Only set sync targets that have an estimated height value [\#115](https://github.com/hyperledger/besu/pull/115)
 - Fix rlpx startup [\#114](https://github.com/hyperledger/besu/pull/114)
@@ -3023,7 +3050,7 @@ For compatibility with ETC Agharta upgrade, use 1.3.7 or later.
 
 ### Technical Improvements
 
-- Less verbose synching subscriptions [\#59](https://github.com/hyperledger/besu/pull/59)
+- Less verbose syncing subscriptions [\#59](https://github.com/hyperledger/besu/pull/59)
 - Return enclave key instead of private transaction hash [\#53](https://github.com/hyperledger/besu/pull/53)
 - Fix mark sweep pruner bugs where nodes that should be kept were being swept  [\#50](https://github.com/hyperledger/besu/pull/50)
 - Clean up BesuConfiguration construction [\#51](https://github.com/hyperledger/besu/pull/51)
@@ -3156,7 +3183,7 @@ For compatibility with ETC Agharta upgrade, use 1.3.7 or later.
 - Updating Orion to v1.3.2 [#1805](https://github.com/PegaSysEng/pantheon/pull/1805)
 - Updaated newHeads subscription to emit events only for canonical blocks [#1798](https://github.com/PegaSysEng/pantheon/pull/1798)
 - Repricing for trie-size-dependent opcodes [#1795](https://github.com/PegaSysEng/pantheon/pull/1795)
-- Revised Istanbul Versioning assignemnts [#1794](https://github.com/PegaSysEng/pantheon/pull/1794)
+- Revised Istanbul Versioning assignments [#1794](https://github.com/PegaSysEng/pantheon/pull/1794)
 - Updated RevertReason to return BytesValue [#1793](https://github.com/PegaSysEng/pantheon/pull/1793)
 - Updated way priv_getPrivacyPrecompileAddress source [#1786](https://github.com/PegaSysEng/pantheon/pull/1786) (thanks to [iikirilov](https://github.com/iikirilov))
 - Updated Chain ID opcode to return 0 as default [#1785](https://github.com/PegaSysEng/pantheon/pull/1785)
@@ -4217,7 +4244,7 @@ has been updated to use the moved quickstart.
 - Fixed deprecation warnings [\#596](https://github.com/PegaSysEng/pantheon/pull/596)
 - IBFT Integration Tests - Future Height [\#591](https://github.com/PegaSysEng/pantheon/pull/591)
 - Added `getNodeData` to `EthPeer` to enable requesting node data [\#589](https://github.com/PegaSysEng/pantheon/pull/589)
-- `Blockcreator` to use `parentblock` specified at constuction [\#588](https://github.com/PegaSysEng/pantheon/pull/588)
+- `Blockcreator` to use `parentblock` specified at construction [\#588](https://github.com/PegaSysEng/pantheon/pull/588)
 - Support responding to `GetNodeData` requests [\#587](https://github.com/PegaSysEng/pantheon/pull/587)
 - IBFT validates block on proposal reception [\#583](https://github.com/PegaSysEng/pantheon/pull/583)
 - Rework `NewRoundValidator` tests [\#582](https://github.com/PegaSysEng/pantheon/pull/582)

README.md

@@ -1,6 +1,6 @@
 # Besu Ethereum Client
  [![CircleCI](https://circleci.com/gh/hyperledger/besu/tree/main.svg?style=svg)](https://circleci.com/gh/hyperledger/besu/tree/main)
- [![Documentation Status](https://readthedocs.org/projects/hyperledger-besu/badge/?version=latest)](https://besu.hyperledger.org/en/latest/?badge=latest)
+ [![Documentation](https://img.shields.io/github/actions/workflow/status/hyperledger/besu-docs/publish-main-docs.yml?branch=main&label=docs)](https://github.com/hyperledger/besu-docs/actions/workflows/publish-main-docs.yml)
  [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/3174/badge)](https://bestpractices.coreinfrastructure.org/projects/3174)
  [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/hyperledger/besu/blob/main/LICENSE)
  [![Discord](https://img.shields.io/discord/905194001349627914?logo=Hyperledger&style=plastic)](https://discord.gg/hyperledger)

acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/node/ProcessBesuNodeRunner.java

@@ -166,6 +166,9 @@ public void startNode(final BesuNode node) {
       if (node.getPrivacyParameters().isPrivacyPluginEnabled()) {
         params.add("--Xprivacy-plugin-enabled");
       }
+      if (node.getPrivacyParameters().isPrivateNonceAlwaysIncrementsEnabled()) {
+        params.add("privacy-nonce-always-increments");
+      }
     }
 
     if (!node.getBootnodes().isEmpty()) {

acceptance-tests/dsl/src/main/java/org/hyperledger/besu/tests/acceptance/dsl/node/configuration/BesuNodeFactory.java

@@ -287,7 +287,8 @@ public BesuNode createNodeWithMultiTenantedPrivacy(
       final String enclaveUrl,
       final String authFile,
       final String privTransactionSigningKey,
-      final boolean enableFlexiblePrivacy)
+      final boolean enableFlexiblePrivacy,
+      final boolean enablePrivateNonceAlwaysIncrements)
       throws IOException, URISyntaxException {
     final PrivacyParameters.Builder privacyParametersBuilder = new PrivacyParameters.Builder();
     final PrivacyParameters privacyParameters =
@@ -298,6 +299,7 @@ public BesuNode createNodeWithMultiTenantedPrivacy(
             .setStorageProvider(new InMemoryPrivacyStorageProvider())
             .setEnclaveFactory(new EnclaveFactory(Vertx.vertx()))
             .setEnclaveUrl(URI.create(enclaveUrl))
+            .setPrivateNonceAlwaysIncrementsEnabled(enablePrivateNonceAlwaysIncrements)
             .setPrivateKeyPath(
                 Paths.get(ClassLoader.getSystemResource(privTransactionSigningKey).toURI()))
             .build();

acceptance-tests/tests/src/test/java/org/hyperledger/besu/tests/acceptance/crypto/SECP256R1AcceptanceTest.java

@@ -66,6 +66,12 @@ public void setUp() throws Exception {
         besu.createNodeWithNonDefaultSignatureAlgorithm(
             "otherNode", GENESIS_FILE, otherNodeKeyPair, List.of(minerNode));
     noDiscoveryCluster.addNode(otherNode);
+
+    minerNode.verify(net.awaitPeerCount(1));
+    otherNode.verify(net.awaitPeerCount(1));
+
+    final var minerChainHead = minerNode.execute(ethTransactions.block());
+    otherNode.verify(blockchain.minimumHeight(minerChainHead.getNumber().longValue()));
   }
 
   @Test

acceptance-tests/tests/src/test/java/org/hyperledger/besu/tests/acceptance/permissioning/NodeSmartContractPermissioningAcceptanceTest.java

@@ -17,6 +17,7 @@
 import org.hyperledger.besu.tests.acceptance.dsl.node.Node;
 
 import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Disabled;
 import org.junit.jupiter.api.Test;
 
 public class NodeSmartContractPermissioningAcceptanceTest
@@ -37,17 +38,17 @@ public void setUp() {
     permissionedCluster.start(bootnode, forbiddenNode, allowedNode, permissionedNode);
 
     // updating permissioning smart contract with allowed nodes
-
+    permissionedNode.verify(nodeIsForbidden(bootnode));
     permissionedNode.execute(allowNode(bootnode));
     permissionedNode.verify(nodeIsAllowed(bootnode));
+    permissionedNode.verify(admin.hasPeer(bootnode));
 
     permissionedNode.execute(allowNode(allowedNode));
     permissionedNode.verify(nodeIsAllowed(allowedNode));
 
     permissionedNode.execute(allowNode(permissionedNode));
     permissionedNode.verify(nodeIsAllowed(permissionedNode));
 
-    permissionedNode.verify(admin.addPeer(bootnode));
     permissionedNode.verify(admin.addPeer(allowedNode));
 
     allowedNode.verify(eth.syncingStatus(false));
@@ -57,6 +58,7 @@ public void setUp() {
   }
 
   @Test
+  @Disabled("test is flaky")
   public void permissionedNodeShouldPeerOnlyWithAllowedNodes() {
     bootnode.verify(net.awaitPeerCount(3));
     allowedNode.verify(net.awaitPeerCount(3));

acceptance-tests/tests/src/test/java/org/hyperledger/besu/tests/acceptance/permissioning/NodeSmartContractPermissioningV2AcceptanceTest.java

@@ -17,6 +17,7 @@
 import org.hyperledger.besu.tests.acceptance.dsl.node.Node;
 
 import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Disabled;
 import org.junit.jupiter.api.Test;
 
 public class NodeSmartContractPermissioningV2AcceptanceTest
@@ -51,6 +52,7 @@ public void setUp() {
   }
 
   @Test
+  @Disabled("test is flaky")
   public void permissionedNodeShouldPeerOnlyWithAllowedNodes() {
     bootnode.verify(net.awaitPeerCount(3));
     allowedNode.verify(net.awaitPeerCount(3));
@@ -60,7 +62,7 @@ public void permissionedNodeShouldPeerOnlyWithAllowedNodes() {
 
   @Test
   public void permissionedNodeShouldDisconnectFromNodeNotPermittedAnymore() {
-    permissionedNode.verify(admin.addPeer(bootnode));
+    permissionedNode.verify(admin.hasPeer(bootnode));
     permissionedNode.verify(admin.addPeer(allowedNode));
     permissionedNode.verify(net.awaitPeerCount(2));
 
@@ -72,7 +74,7 @@ public void permissionedNodeShouldDisconnectFromNodeNotPermittedAnymore() {
 
   @Test
   public void permissionedNodeShouldConnectToNewlyPermittedNode() {
-    permissionedNode.verify(admin.addPeer(bootnode));
+    permissionedNode.verify(admin.hasPeer(bootnode));
     permissionedNode.verify(admin.addPeer(allowedNode));
     permissionedNode.verify(net.awaitPeerCount(2));
 
@@ -87,7 +89,7 @@ public void permissionedNodeShouldConnectToNewlyPermittedNode() {
 
   @Test
   public void permissioningUpdatesPropagateThroughNetwork() {
-    permissionedNode.verify(admin.addPeer(bootnode));
+    permissionedNode.verify(admin.hasPeer(bootnode));
     permissionedNode.verify(admin.addPeer(allowedNode));
     permissionedNode.verify(net.awaitPeerCount(2));
 

acceptance-tests/tests/src/test/java/org/hyperledger/besu/tests/acceptance/permissioning/NodeSmartContractPermissioningV2DNSAcceptanceTest.java

@@ -25,8 +25,10 @@
 
 import org.assertj.core.api.Assertions;
 import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Disabled;
 import org.junit.jupiter.api.Test;
 
+@Disabled("test is flaky #7191")
 public class NodeSmartContractPermissioningV2DNSAcceptanceTest
     extends NodeSmartContractPermissioningV2AcceptanceTestBase {
 
@@ -57,12 +59,6 @@ public void setUp() {
 
     permissionedNode.execute(allowNode(permissionedNode));
     permissionedNode.verify(connectionIsAllowed(permissionedNode));
-
-    // Verify initial configuration
-    bootnode.verify(net.awaitPeerCount(3));
-    allowedNode.verify(net.awaitPeerCount(3));
-    forbiddenNode.verify(net.awaitPeerCount(2));
-    permissionedNode.verify(net.awaitPeerCount(2));
   }
 
   @Test

acceptance-tests/tests/src/test/java/org/hyperledger/besu/tests/acceptance/privacy/multitenancy/MultiTenancyAcceptanceTest.java

@@ -104,6 +104,7 @@ public void setUp() throws Exception {
             "http://127.0.0.1:" + wireMockRule.port(),
             "authentication/auth_priv.toml",
             "authentication/auth_priv_key",
+            false,
             false);
     multiTenancyCluster.start(node);
     final String token =