receiving random grpc._channel._MultiThreadedRendezvous errors from iroha server every 24hours or so

[turbopape]

Hello,
[I already posted this to the iroha main repo]
We are using Iroha 1.2 latest on kubernetes. Setup is a single peer, with a postgresql 13 DB - and under like 20 transactions per second.
We are using the python client. We are streaming through the results using a generator. here is the code:

def send_tx_and_stream_result(net, tx, user_private_key):
    iroha.IrohaCrypto.sign_transaction(tx, user_private_key)
    hex_hash = binascii.hexlify(iroha.IrohaCrypto.hash(tx))
    print('Transaction hash = {}, creator = {}'.format(
        hex_hash, tx.payload.reduced_payload.creator_account_id))
    net.send_tx(tx)
    for status in net.tx_status_stream(tx, timeout=args.wlts_iroha_tx_status_timeoout):
        txStatusResponse = wallets_pb2.TxStatusResponse()
        txStatusResponse.status_name = status[0]
        txStatusResponse.status_code = status[1]
        txStatusResponse.error_code = status[2]
        yield txStatusResponse

Iroha works fine and then, every now and then, we start seeing the message:
'grpc._channel._MultiThreadedRendezvous ' errors and Iroha stops being responsive.

Here is our configmap and genesys:

apiVersion: v1
kind: ConfigMap
metadata:
  name: iroha-config
  namespace: caravel-iroha
data:
  genesis.block: |
    {
      "block_v1":{
        "payload":{
            "transactions":[
              {
                  "payload":{
                    "reducedPayload":{
                        "commands":[
                          {
                              "addPeer":{
                                "peer":{
                                    "address":"XYZ:10001",
                                    "peerKey":"XYZ"
                                }
                              }
                          },
                          {
                              "createRole":{
                                "roleName":"admin",
                                "permissions":[
                                    "can_add_peer",
                                    "can_add_signatory",
                                    "can_create_account",
                                    "can_create_domain",
                                    "can_get_all_acc_ast",
                                    "can_get_all_acc_ast_txs",
                                    "can_get_all_acc_detail",
                                    "can_get_all_acc_txs",
                                    "can_get_all_accounts",
                                    "can_get_all_signatories",
                                    "can_get_all_txs",
                                    "can_get_blocks",
                                    "can_get_roles",
                                    "can_read_assets",
                                    "can_remove_signatory",
                                    "can_set_quorum",
                                    "can_receive"
                                ]
                              }
                          },
                          {
                              "createRole":{
                                "roleName":"user",
                                "permissions":[
                                    "can_add_signatory",
                                    "can_get_my_acc_ast",
                                    "can_get_my_acc_ast_txs",
                                    "can_get_my_acc_detail",
                                    "can_get_my_acc_txs",
                                    "can_get_my_account",
                                    "can_get_my_signatories",
                                    "can_get_my_txs",
                                    "can_grant_can_add_my_signatory",
                                    "can_grant_can_remove_my_signatory",
                                    "can_grant_can_set_my_account_detail",
                                    "can_grant_can_set_my_quorum",
                                    "can_grant_can_transfer_my_assets",
                                    "can_receive",
                                    "can_remove_signatory",
                                    "can_set_quorum",
                                    "can_transfer"
                                ]
                              }
                          },
                          {
                              "createRole":{
                                "roleName":"money_creator",
                                "permissions":[
                                    "can_add_asset_qty",
                                    "can_create_asset",
                                    "can_receive",
                                    "can_transfer"
                                ]
                              }
                          },
                          {
                              "createDomain":{
                                "domainId":"caravel",
                                "defaultRole":"user"
                              }
                          },
                          {
                              "createAsset":{
                                "assetName":"XYZ",
                                "domainId":"XYZ",
                                "precision":2
                              }
                          },
                          {
                              "createAccount":{
                                "accountName":"XYZ",
                                "domainId":"XYZ",
                                "publicKey":"XYZ"
                              }
                          },
                          {
                              "createAccount":{
                                "accountName":"XYZ",
                                "domainId":"XYZ",
                                "publicKey":"XYZ"
                              }
                          },
                          {
                              "createAccount":{
                                "accountName":"XYZ",
                                "domainId":"XYZ",
                                "publicKey":"XYZ"
                              }
                          },
                        
                          {
                              "appendRole":{
                                "accountId":"XYZ@XYZ",
                                "roleName":"admin"
                              }
                          },
                          {
                              "appendRole":{
                                "accountId":"XYZ@XYZ",
                                "roleName":"money_creator"
                              }
                          }
                        ],
                        "quorum":1
                    }
                  }
              }
            ],
            "txNumber":1,
            "height":"1",
            "prevBlockHash":"0000000000000000000000000000000000000000000000000000000000000000"
        }
      }
    }
  config.docker: |
    {
      "block_store_path" : "/opt/block_store/",
      "torii_port" : 50051,
      "internal_port" : 10001,
      "pg_opt" : "host=XYZ port=5432 user=XYZ password=XYZ dbname=irohadefault",
      "database": {
        "host": "XYZ",
        "port": 5432,
        "user": "XYZ",
        "password": "XYZ",
        "working database": "iroha_data",
        "maintenance database": "iroha_maintenance"
      },
      "max_proposal_size" : 3000,
      "proposal_delay" : 1000,
      "vote_delay" : 5000,
      "load_delay" : 5000,
      "mst_enable" : true,
      "log": {
        "level": "trace",
        "patterns": {
          "debug": "debug %v",
          "error": "error %v!"
        },
        "children": {
          "KeysManager": {
            "level": "trace"
          },
          "Irohad": {
            "children": {
              "Storage": {
                "level": "trace",
                "patterns": {
                  "debug": "thread %t: %v."
                }
              }
            }
          }
        }
      }
    }

Do you think me using the generator being a problem under heavy load ?
Thank you so much,
Rafik

[baziorek]

The issue is somehow connected with: #48