Skip to content

receiving random grpc._channel._MultiThreadedRendezvous errors from iroha server every 24hours or so #79

Open
@turbopape

Description

@turbopape

Hello,
[I already posted this to the iroha main repo]
We are using Iroha 1.2 latest on kubernetes. Setup is a single peer, with a postgresql 13 DB - and under like 20 transactions per second.
We are using the python client. We are streaming through the results using a generator. here is the code:

def send_tx_and_stream_result(net, tx, user_private_key):
    iroha.IrohaCrypto.sign_transaction(tx, user_private_key)
    hex_hash = binascii.hexlify(iroha.IrohaCrypto.hash(tx))
    print('Transaction hash = {}, creator = {}'.format(
        hex_hash, tx.payload.reduced_payload.creator_account_id))
    net.send_tx(tx)
    for status in net.tx_status_stream(tx, timeout=args.wlts_iroha_tx_status_timeoout):
        txStatusResponse = wallets_pb2.TxStatusResponse()
        txStatusResponse.status_name = status[0]
        txStatusResponse.status_code = status[1]
        txStatusResponse.error_code = status[2]
        yield txStatusResponse

Iroha works fine and then, every now and then, we start seeing the message:
'grpc._channel._MultiThreadedRendezvous ' errors and Iroha stops being responsive.

Here is our configmap and genesys:

apiVersion: v1
kind: ConfigMap
metadata:
  name: iroha-config
  namespace: caravel-iroha
data:
  genesis.block: |
    {
      "block_v1":{
        "payload":{
            "transactions":[
              {
                  "payload":{
                    "reducedPayload":{
                        "commands":[
                          {
                              "addPeer":{
                                "peer":{
                                    "address":"XYZ:10001",
                                    "peerKey":"XYZ"
                                }
                              }
                          },
                          {
                              "createRole":{
                                "roleName":"admin",
                                "permissions":[
                                    "can_add_peer",
                                    "can_add_signatory",
                                    "can_create_account",
                                    "can_create_domain",
                                    "can_get_all_acc_ast",
                                    "can_get_all_acc_ast_txs",
                                    "can_get_all_acc_detail",
                                    "can_get_all_acc_txs",
                                    "can_get_all_accounts",
                                    "can_get_all_signatories",
                                    "can_get_all_txs",
                                    "can_get_blocks",
                                    "can_get_roles",
                                    "can_read_assets",
                                    "can_remove_signatory",
                                    "can_set_quorum",
                                    "can_receive"
                                ]
                              }
                          },
                          {
                              "createRole":{
                                "roleName":"user",
                                "permissions":[
                                    "can_add_signatory",
                                    "can_get_my_acc_ast",
                                    "can_get_my_acc_ast_txs",
                                    "can_get_my_acc_detail",
                                    "can_get_my_acc_txs",
                                    "can_get_my_account",
                                    "can_get_my_signatories",
                                    "can_get_my_txs",
                                    "can_grant_can_add_my_signatory",
                                    "can_grant_can_remove_my_signatory",
                                    "can_grant_can_set_my_account_detail",
                                    "can_grant_can_set_my_quorum",
                                    "can_grant_can_transfer_my_assets",
                                    "can_receive",
                                    "can_remove_signatory",
                                    "can_set_quorum",
                                    "can_transfer"
                                ]
                              }
                          },
                          {
                              "createRole":{
                                "roleName":"money_creator",
                                "permissions":[
                                    "can_add_asset_qty",
                                    "can_create_asset",
                                    "can_receive",
                                    "can_transfer"
                                ]
                              }
                          },
                          {
                              "createDomain":{
                                "domainId":"caravel",
                                "defaultRole":"user"
                              }
                          },
                          {
                              "createAsset":{
                                "assetName":"XYZ",
                                "domainId":"XYZ",
                                "precision":2
                              }
                          },
                          {
                              "createAccount":{
                                "accountName":"XYZ",
                                "domainId":"XYZ",
                                "publicKey":"XYZ"
                              }
                          },
                          {
                              "createAccount":{
                                "accountName":"XYZ",
                                "domainId":"XYZ",
                                "publicKey":"XYZ"
                              }
                          },
                          {
                              "createAccount":{
                                "accountName":"XYZ",
                                "domainId":"XYZ",
                                "publicKey":"XYZ"
                              }
                          },
                        
                          {
                              "appendRole":{
                                "accountId":"XYZ@XYZ",
                                "roleName":"admin"
                              }
                          },
                          {
                              "appendRole":{
                                "accountId":"XYZ@XYZ",
                                "roleName":"money_creator"
                              }
                          }
                        ],
                        "quorum":1
                    }
                  }
              }
            ],
            "txNumber":1,
            "height":"1",
            "prevBlockHash":"0000000000000000000000000000000000000000000000000000000000000000"
        }
      }
    }
  config.docker: |
    {
      "block_store_path" : "/opt/block_store/",
      "torii_port" : 50051,
      "internal_port" : 10001,
      "pg_opt" : "host=XYZ port=5432 user=XYZ password=XYZ dbname=irohadefault",
      "database": {
        "host": "XYZ",
        "port": 5432,
        "user": "XYZ",
        "password": "XYZ",
        "working database": "iroha_data",
        "maintenance database": "iroha_maintenance"
      },
      "max_proposal_size" : 3000,
      "proposal_delay" : 1000,
      "vote_delay" : 5000,
      "load_delay" : 5000,
      "mst_enable" : true,
      "log": {
        "level": "trace",
        "patterns": {
          "debug": "debug %v",
          "error": "error %v!"
        },
        "children": {
          "KeysManager": {
            "level": "trace"
          },
          "Irohad": {
            "children": {
              "Storage": {
                "level": "trace",
                "patterns": {
                  "debug": "thread %t: %v."
                }
              }
            }
          }
        }
      }
    }

Do you think me using the generator being a problem under heavy load ?
Thank you so much,
Rafik

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions