[PoA] Transactions are not propagated when adding a new node to the network

[fooock]

Before filing a new issue, please provide the following information.

• Parity Ethereum version: Parity-Ethereum/v2.1.8-stable-3eae1d3-20181130/x86_64-linux-gnu/rustc1.30.1
• Operating system: Linux
• Installation: one-line installer
• Fully synchronized: yes
• Network: Private PoA
• Restarted: yes

I'm trying the demo PoA tutorial. All seems to work, but when I add a new node to the network (this node isn't validator), the transactions from this node doesn't propagate to the other nodes in the network.

The initial network is composed of the nodes:

• validator-01
• validator-02

The updated network is with the node:

• regular-02

All transactions related to this issue are from regular-02 node.

The genesis file for my network is:

{
  "name":"primer",
  "engine":{
    "authorityRound":{
      "params":{
        "stepDuration":"5",
        "blockReward":"0xDE0B6B3A7640000",
        "validators":{
          "list":[
            "0x001b3fc96e38936badeb3b43bf8b05e43c6128e3",
            "0x009dfd994dfe87af895bd9a6a3e5576182d52d9f"
          ]
        }
      }
    }
  },
  "params":{
    "gasLimitBoundDivisor":"0x400",
    "maximumExtraDataSize":"0x20",
    "minGasLimit":"0x1388",
    "networkID":"0x3144E",
    "eip155Transition":0,
    "validateChainIdTransition":0,
    "eip140Transition":0,
    "eip211Transition":0,
    "eip214Transition":0,
    "eip658Transition":0
  },
  "genesis":{
    "seal":{
      "authorityRound":{
        "step":"0x0",
        "signature":"0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
      }
    },
    "difficulty":"0x20000",
    "gasLimit":"0x5B8D80"
  },
  "accounts":{
    "0x0000000000000000000000000000000000000001":{
      "balance":"1",
      "builtin":{
        "name":"ecrecover",
        "pricing":{
          "linear":{
            "base":3000,
            "word":0
          }
        }
      }
    },
    "0x0000000000000000000000000000000000000002":{
      "balance":"1",
      "builtin":{
        "name":"sha256",
        "pricing":{
          "linear":{
            "base":60,
            "word":12
          }
        }
      }
    },
    "0x0000000000000000000000000000000000000003":{
      "balance":"1",
      "builtin":{
        "name":"ripemd160",
        "pricing":{
          "linear":{
            "base":600,
            "word":120
          }
        }
      }
    },
    "0x0000000000000000000000000000000000000004":{
      "balance":"1",
      "builtin":{
        "name":"identity",
        "pricing":{
          "linear":{
            "base":15,
            "word":3
          }
        }
      }
    },
    "0x89f9097168fd14bdf95155f5b268759d8550d43c":{
      "balance":"10000000000000000000000000"
    }
  }
}

The .toml configuration file from the regular node is:

[parity]                                                                                                                                                                                                           
chain = "/parity/primer.json"                                                                                                                                                                                      
base_path = "/parity/regular-02"                                                                                                                                                                                   
identity = "primer-regular-02"                                                                                                                                                                           
no_persistent_txqueue = true                                                                                                                                                                                       
[network]                                                                                                                                                                                                          
port = 30302                                                                                                                                                                                                       
[rpc]                                                                                                                                                                                                              
interface = "all"                                                                                                                                                                                                  
port = 8542                                                                                                                                                                                                        
cors = ["*"]                                                                                                                                                                                                       
apis = ["web3", "eth", "net", "personal", "parity", "parity_set", "traces", "rpc", "parity_accounts"]                                                                                                              
[ui]                                                                                                                                                                                                               
port = 8182                                                                                                                                                                                                        
[websockets]                                                                                                                                                                                                       
port = 8452                                                                                                                                                                                                        
[ipc]                                                                                                                                                                                                              
disable = true 

The .toml config file from one of the validator nodes is (the other one is the same config but with different identity and base path):

[parity]                                                                                                                                                                                                           
chain = "/parity/primer.json"                                                                                                                                                                                      
base_path = "/parity/validator-02"                                                                                                                                                                                 
identity = "primer-validator-02"                                                                                                                                                                         
no_persistent_txqueue = true                                                                                                                                                                                       
[network]                                                                                                                                                                                                          
port = 30300                                                                                                                                                                                                       
min_peers = 50                                                                                                                                                                                                     
max_peers = 100                                                                                                                                                                                                    
[rpc]                                                                                                                                                                                                              
port = 8540                                                                                                                                                                                                        
cors = ["*"]                                                                                                                                                                                                       
apis = ["web3", "eth", "net", "personal", "parity", "parity_set", "traces", "rpc", "parity_accounts"]                                                                                                              
[ui]                                                                                                                                                                                                               
port = 8180                                                                                                                                                                                                        
[websockets]                                                                                                                                                                                                       
port = 8450                                                                                                                                                                                                        
[account]                                                                                                                                                                                                          
password = ["/parity/validator-02/node.pwds"]                                                                                                                                                                      
[mining]                                                                                                                                                                                                           
engine_signer = "0x001b3fc96e38936badeb3b43bf8b05e43c6128e3"                                                                                                                                                       
force_sealing = true                                                                                                                                                                                               
reseal_on_txs = "all"  

When I send a new transaction from this node, a new transaction is created:

curl --data '{"jsonrpc":"2.0","method":"personal_sendTransaction","params":[{\
  "from":"0x00c498303595303a7533c47cf32f632087dc1c0c",\
  "to":"0x009dfd994dfe87af895bd9a6a3e5576182d52d9f",\
  "value":"0xDE0B6B3A7640000"}, "**************"],"id":0}' \
  -H "Content-Type: application/json" -X POST localhost:8542                                                                                             

{"jsonrpc":"2.0","result":"0x3135e61208a47af85e677be1e9f39efbd7a81098212f525f4cbcf6be082e8930","id":0}   

When I want to retrieve the transaction info from this node, I obtain this result:

curl -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionByHash","params":["0x3135e61208a47af85e677be1e9f39efbd7a81098212f525f4cbcf6be082e8930"],"id":1}' -H "Content
-Type: application/json" localhost:8542

{                                                                                                                                                                                                                  
  "jsonrpc": "2.0",
  "result": {
    "blockHash": null,
    "blockNumber": null,
    "chainId": "0x3144e",
    "condition": null,
    "creates": null,
    "from": "0x00c498303595303a7533c47cf32f632087dc1c0c",
    "gas": "0x186a00",
    "gasPrice": "0x0",
    "hash": "0x3135e61208a47af85e677be1e9f39efbd7a81098212f525f4cbcf6be082e8930",
    "input": "0x",
    "nonce": "0x16",
    "publicKey": "0xa0263cf09db59c38b4c155aa72b059b06db26cb7123ef07a48025960254268e45496890d7aa258f1337ea43d9a0d1bb05ee99410f42205c6dd817276a311e512",
    "r": "0xefad4a024b155b4bb83c519ac3f5991184c7579d328f13ce1c0b5c922f7e5b75",
    "raw": "0xf86b168083186a0094009dfd994dfe87af895bd9a6a3e5576182d52d9f880de0b6b3a764000080830628bfa0efad4a024b155b4bb83c519ac3f5991184c7579d328f13ce1c0b5c922f7e5b75a06e97cd60d336c666ba702ba52abc49862f6158c2368
5350c89c79e1c6874560c",
    "s": "0x6e97cd60d336c666ba702ba52abc49862f6158c23685350c89c79e1c6874560c",
    "standardV": "0x0",
    "to": "0x009dfd994dfe87af895bd9a6a3e5576182d52d9f",
    "transactionIndex": null,
    "v": "0x628bf",
    "value": "0xde0b6b3a7640000"
  },
  "id": 1
}

From the Ethereum network status this is what I see:

I expect the transaction sent from the regular-02 node has been mined, but this never occurs.

What I'm missing?

When I use Parity 1.11.7 or 1.11.11, this network config work as expected, and all transactions are mined.

Related?

#9329

[jam10o-new]

Can you share the txqueue logs from your validator nodes? (-l txqueue=trace) when you try getTransactionByHash the result indicates that the transaction is still pending (blockHash is null, blockNumber is null) so that either means the transaction is either not reaching your validation nodes, or not being accepted by them once they reach the txqueue. Do you see the transaction if you try that rpc call on your validation nodes?

[fooock]

txqueue logs from validator nodes here. When I execute the last command to see the transaction from my validator-02 node the response is nothing:

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionByHash","params":["0x3135e61208a47af85e677be1e9f39efbd7a81098212f525f4cbcf6be082e8930"],"id":1}' -
H "Content -Type: application/json" localhost:8540
$

Same for the validator-01 node.

But if I try to view all transactions from my regular node, all pending transactions appear:

$ curl --data '{"method":"parity_allTransactions","params":[],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST localhost:8542 | jq
{
  "jsonrpc": "2.0",
  "result": [
    {}, ......
    {
      "blockHash": null,
      "blockNumber": null,
      "chainId": "0x3144e",
      "condition": null,
      "creates": null,
      "from": "0x00c498303595303a7533c47cf32f632087dc1c0c",
      "gas": "0x186a00",
      "gasPrice": "0x0",
      "hash": "0x2b96e93981db005758e66d04cc3b40e198e16ee87191d793ac9cadf7ddb1b507",
      "input": "0x",
      "nonce": "0x4",
      "publicKey": "0xa0263cf09db59c38b4c155aa72b059b06db26cb7123ef07a48025960254268e45496890d7aa258f1337ea43d9a0d1bb05ee99410f42205c6dd817276a311e512",
      "r": "0xff0e26b13567e5a5d683d88e89d6f0d273331a5e081d69e79e86c131cc2cd0ac",
      "raw": "0xf86b048083186a0094009dfd994dfe87af895bd9a6a3e5576182d52d9f880de0b6b3a764000080830628c0a0ff0e26b13567e5a5d683d88e89d6f0d273331a5e081d69e79e86c131cc2cd0aca0054970f35763fa4152934ad594aadc1e6f096da73
f02ed6735b63d23dd0d24be",
      "s": "0x54970f35763fa4152934ad594aadc1e6f096da73f02ed6735b63d23dd0d24be",
      "standardV": "0x1",
      "to": "0x009dfd994dfe87af895bd9a6a3e5576182d52d9f",
      "transactionIndex": null,
      "v": "0x628c0",
      "value": "0xde0b6b3a7640000"
    },
    {
      "blockHash": null,
      "blockNumber": null,
      "chainId": "0x3144e",
      "condition": null,
      "creates": null,
      "from": "0x00c498303595303a7533c47cf32f632087dc1c0c",
      "gas": "0x186a00",
      "gasPrice": "0x0",
      "hash": "0xeaf88ce8ace42ebecfe966225b13c82d5fe3e47aad465385da6d573a5fa769b9",
      "input": "0x",
      "nonce": "0x5",
      "publicKey": "0xa0263cf09db59c38b4c155aa72b059b06db26cb7123ef07a48025960254268e45496890d7aa258f1337ea43d9a0d1bb05ee99410f42205c6dd817276a311e512",
      "r": "0x9145d84b8fc9da4854d6f6a18f57c9e8b80ee325b1000dab869ab66487e993a9",
      "raw": "0xf86b058083186a0094009dfd994dfe87af895bd9a6a3e5576182d52d9f880de0b6b3a764000080830628c0a09145d84b8fc9da4854d6f6a18f57c9e8b80ee325b1000dab869ab66487e993a9a004dd2d87a058629ec3b7392c01380478ae237cb9a
c5e011f10bc94f688f32c4a",
      "s": "0x4dd2d87a058629ec3b7392c01380478ae237cb9ac5e011f10bc94f688f32c4a",
      "standardV": "0x1",
      "to": "0x009dfd994dfe87af895bd9a6a3e5576182d52d9f",
      "transactionIndex": null,
      "v": "0x628c0",
      "value": "0xde0b6b3a7640000"
    },
    {},.....
    {
      "blockHash": null,
      "blockNumber": null,
      "chainId": "0x3144e",
      "condition": null,
      "creates": null,
      "from": "0x00c498303595303a7533c47cf32f632087dc1c0c",
      "gas": "0x186a00",
      "gasPrice": "0x0",
      "hash": "0xc31e2886ac200c325e20a36036d58d6d832c757e027bd17c0e62a818b07db71e",
      "input": "0x",
      "nonce": "0xa",
      "publicKey": "0xa0263cf09db59c38b4c155aa72b059b06db26cb7123ef07a48025960254268e45496890d7aa258f1337ea43d9a0d1bb05ee99410f42205c6dd817276a311e512",
      "r": "0xb0e745ec14a47a62f548df4dbfa8c148d9ed08b0fb28e4dde8d876e41c3d6aa9",
      "raw": "0xf86b0a8083186a0094009dfd994dfe87af895bd9a6a3e5576182d52d9f880de0b6b3a764000080830628c0a0b0e745ec14a47a62f548df4dbfa8c148d9ed08b0fb28e4dde8d876e41c3d6aa9a00d4264a679f14f6a5150706be81bc1b6b33841b91
c59de7e3e75ffb54e31e8ef",
      "s": "0xd4264a679f14f6a5150706be81bc1b6b33841b91c59de7e3e75ffb54e31e8ef",
      "standardV": "0x1",
      "to": "0x009dfd994dfe87af895bd9a6a3e5576182d52d9f",
      "transactionIndex": null,
      "v": "0x628c0",
      "value": "0xde0b6b3a7640000"
    },
    {},.......
    {
      "blockHash": null,
      "blockNumber": null,
      "chainId": "0x3144e",
      "condition": null,
      "creates": null,
      "from": "0x00c498303595303a7533c47cf32f632087dc1c0c",
      "gas": "0x186a00",
      "gasPrice": "0x0",
      "hash": "0xe8ac167d33ec8d5aed947d2bea63cc5cf8308c0337b620ec530339ccd3c872ea",
      "input": "0x",
      "nonce": "0x15",
      "publicKey": "0xa0263cf09db59c38b4c155aa72b059b06db26cb7123ef07a48025960254268e45496890d7aa258f1337ea43d9a0d1bb05ee99410f42205c6dd817276a311e512",
      "r": "0x4520d4894b1af5c20ba09897edd39d0a4cb19e8b37e6fb54d51d4affd0b058c4",
      "raw": "0xf86b158083186a0094009dfd994dfe87af895bd9a6a3e5576182d52d9f880de0b6b3a764000080830628bfa04520d4894b1af5c20ba09897edd39d0a4cb19e8b37e6fb54d51d4affd0b058c4a00e59571009ee79de483e904f635506e6097f0e454
f8d65e2a6d0afdb0739c91d",
      "s": "0xe59571009ee79de483e904f635506e6097f0e454f8d65e2a6d0afdb0739c91d",
      "standardV": "0x0",
      "to": "0x009dfd994dfe87af895bd9a6a3e5576182d52d9f",
      "transactionIndex": null,
      "v": "0x628bf",
      "value": "0xde0b6b3a7640000"
    },
    {
      "blockHash": null,
      "blockNumber": null,
      "chainId": "0x3144e",
      "condition": null,
      "creates": null,
      "from": "0x00c498303595303a7533c47cf32f632087dc1c0c",
      "gas": "0x186a00",
      "gasPrice": "0x0",
      "hash": "0x3135e61208a47af85e677be1e9f39efbd7a81098212f525f4cbcf6be082e8930",
      "input": "0x",
      "nonce": "0x16",
      "publicKey": "0xa0263cf09db59c38b4c155aa72b059b06db26cb7123ef07a48025960254268e45496890d7aa258f1337ea43d9a0d1bb05ee99410f42205c6dd817276a311e512",
      "r": "0xefad4a024b155b4bb83c519ac3f5991184c7579d328f13ce1c0b5c922f7e5b75",
      "raw": "0xf86b168083186a0094009dfd994dfe87af895bd9a6a3e5576182d52d9f880de0b6b3a764000080830628bfa0efad4a024b155b4bb83c519ac3f5991184c7579d328f13ce1c0b5c922f7e5b75a06e97cd60d336c666ba702ba52abc49862f6158c23
685350c89c79e1c6874560c",
      "s": "0x6e97cd60d336c666ba702ba52abc49862f6158c23685350c89c79e1c6874560c",
      "standardV": "0x0",
      "to": "0x009dfd994dfe87af895bd9a6a3e5576182d52d9f",
      "transactionIndex": null,
      "v": "0x628bf",
      "value": "0xde0b6b3a7640000"
    }
  ],
  "id": 1
}  

[jam10o-new]

Yeah, you're certainly correct that the transactions don't seem to be being propagated in the first place. Your netstats also show your nodes are all in sync and on the same chain, that rules out connectivity issues. I'm seeing other users with similar issues elsewhere.

It looks like you're running with --no-persistent-txqueue. Have the transactions (with lower nonces) before those on the txqueue been confirmed already? This is likely a case of nodes not accepting transactions because they don't have the dependencies of the transaction.

[fooock]

No transactions from any validator node:

parity-validator-01

$ curl --data '{"method":"parity_allTransactions","params":[],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST localhost:8540
{"jsonrpc":"2.0","result":[],"id":1}

Using the address from my validator-01 node the client return 3 pending transactions:

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0x009dfd994dfe87af895bd9a6a3e5576182d52d9f","pending"],"id":1}' -H "Content-Type
: application/json" localhost:8540
                                                                                                                                                                                
{"jsonrpc":"2.0","result":"0x3","id":1}  

Same number of transactions using the latest param.

parity-validator-02

$ curl --data '{"method":"parity_allTransactions","params":[],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST localhost:8540
{"jsonrpc":"2.0","result":[],"id":1}

No transactions sent using this validator.

What do you mean by dependencies of the transaction? The transactions I created are to move funds between accounts and no external dependencies are involved

[jam10o-new]

ie, transaction nonces

[fooock]

No transactions confirmed from the regular node. The pending transactions shown in one of my last comments is shorted to not occupy much space, currently, nonces from these transactions are from 0x0 to 0x16

[tomusdrw]

@fooock:

1. could you run the the regular node with -lsync=trace,txpool=trace?
2. Seems that you are sending more than 16 transactions from the same address, can you check if setting --txqueue-per-sender 1024 on the regular node changes anything (related Local transactions should be accepted over the limit #9909 )
3. Is regular node connected to validators and on the same block?
4. Could you run parity_nextNonce("0x00c498303595303a7533c47cf32f632087dc1c0c") and eth_getTransactionCount("0x00c498303595303a7533c47cf32f632087dc1c0c", "latest") on the regular node and also on validator nodes?
5. The validator logs never have anything in the pool, which seems suspicious - probably the transactions are not sent at all.

[fooock]

1 - Logs for the regular node can be found here
3 - Yes, regular node is connected to two nodes
4 - Results:

regular-02

• Method parity_nextNonce:

$ curl -X POST --data '{"jsonrpc":"2.0","method":"parity_nextNonce","params":["0x00c498303595303a7533c47cf32f632087dc1c0c"],"id":1}' -H "Content-Type: application/json" localhost:8542
{"jsonrpc":"2.0","result":"0x0","id":1} 

• Method eth_getTransactionCount:

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0x00c498303595303a7533c47cf32f632087dc1c0c", "latest"],"id":1}' -H "Content-Type: application/json" localhost:8542                                                             
{"jsonrpc":"2.0","result":"0x0","id":1}  

validator-01

• Method parity_nextNonce:

$ curl -X POST --data '{"jsonrpc":"2.0","method":"parity_nextNonce","params":["0x00c498303595303a7533c47cf32f632087dc1c0c"],"id":1}' -H "Content-Type: application/json" localhost:8540                                                                                   
{"jsonrpc":"2.0","result":"0x0","id":1}

• Method eth_getTransactionCount:

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0x00c498303595303a7533c47cf32f632087dc1c0c", "latest"],"id":1}' -H "Content-Type: application/json" localhost:8540                                                             
{"jsonrpc":"2.0","result":"0x0","id":1}  

validator-02

• Method parity_nextNonce:

$ curl -X POST --data '{"jsonrpc":"2.0","method":"parity_nextNonce","params":["0x00c498303595303a7533c47cf32f632087dc1c0c"],"id":1}' -H "Content-Type: application/json" localhost:8540                                                                                   
{"jsonrpc":"2.0","result":"0x0","id":1}

• Method eth_getTransactionCount:

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_getTransactionCount","params":["0x00c498303595303a7533c47cf32f632087dc1c0c", "latest"],"id":1}' -H "Content-Type: application/json" localhost:8540                                                             
{"jsonrpc":"2.0","result":"0x0","id":1}  

I will update the configuration to add the --txqueue-per-sender 1024 flag into the regular node.

[fooock]

Using --tx-queue-per-sender 1024 doesn't changes anything

[tomusdrw]

@fooock thank you so much for the logs and results, it seems thought that your transaction pool is now empty and you don't have any transactions from 0x00c498303595303a7533c47cf32f632087dc1c0c, can you confirm that?

Would you mind reproducing the situation again and provide the same logs in the case where transactions are there?