bad performance on mainnet eth api pressure test

[TonyS001]

System information

java-tron version: 4.7.1.1
OS & Version: Linux
Commit hash : d5da6ae
JVM: -Xms8g -Xmx16g -XX:+UseConcMarkSweepGC -XX:+PrintGCDetails -Xloggc:logs/gc.log -XX:+PrintGCDateStamps -XX:+CMSParallelRemarkEnabled -XX:ReservedCodeCacheSize=256m -XX:+UseCodeCacheFlushing -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m -XX:MaxDirectMemorySize=1g -XX:+HeapDumpOnOutOfMemoryError -XX:NewRatio=2

Actual behaviour

I did a pressure test on mainnet fullnode by constantly sending eth_blockNumber query.
Results show the response time gets higher and higher.
TPS is only 1000/s which is far below expectations.
Many requests failed especially during the second half of the test.

Backtrace

I attach the log file during the test.
I want to know which parameters should I mainly checked in the log.
Looking forward to your reply! Thanks!

test.log

[m-o-m12]

@TonyS001 Every api has access rate limit. The default rate limit is 1000 per seconds. You can customize qps for each API through configuration items.rate.limiter.http for http and json-rpc APIs and rate.limiter.rpc for grpc APIs. Please refer to below configure items. For http&grpc API, we should configure for each http/grpc api. but for json-rpc api, only a JsonRpcServlet can limit all json-rpc APIs.

rate.limiter = {
  http = [
    #  Preemptive: the number of concurrent accesses to the interface
    #  {
    #    component = "GetNowBlockServlet",
    #    strategy = "GlobalPreemptibleAdapter",
    #    paramString = "permit=1"
    #  },
    #  The maximum number of accesses per second for an IP on the interface
    #  {
    #    component = "GetAccountServlet",
    #    strategy = "IPQPSRateLimiterAdapter",
    #    paramString = "qps=1"
    #  },
    #  The maximum number of accesses per second on the interface
    #  {
    #    component = "JsonRpcServlet",
    #    strategy = "QpsRateLimiterAdapter",
    #    paramString = "qps=2000"
    #  }
  ],

  rpc = [
    #  {
    #    component = "protocol.Wallet/GetBlockByLatestNum2",
    #    strategy = "GlobalPreemptibleAdapter",
    #    paramString = "permit=1"
    #  },

    #  {
    #    component = "protocol.Wallet/GetAccount",
    #    strategy = "IPQPSRateLimiterAdapter",
    #    paramString = "qps=1"
    #  },

    #  {
    #    component = "protocol.Wallet/ListWitnesses",
    #    strategy = "QpsRateLimiterAdapter",
    #    paramString = "qps=1"
    #  },
  ]
}

[TonyS001]

@TonyS001 Every api has access rate limit. The default rate limit is 1000 per seconds. You can customize qps for each API through configuration items.rate.limiter.http for http and json-rpc APIs and rate.limiter.rpc for grpc APIs. Please refer to below configure items. For http&grpc API, we should configure for each http/grpc api. but for json-rpc api, only a JsonRpcServlet can limit all json-rpc APIs.

Thanks a lot.

rate.limiter = {
  http = [
       {
         component = "JsonRpcServlet",
         strategy = "QpsRateLimiterAdapter",
         paramString = "qps=20000"
       }
  ],
  rpc = [
  ]
}

I set config as above. But the test result is still bad.
I wonder if I should both set IPQPSRateLimiterAdapte and QpsRateLimiterAdapter.
I notice QpsRateLimiterAdapter has a default set DefaultBaseQqsAdapter("qps=1000"), what about IPQPSRateLimiterAdapte? If I only set QpsRateLimiterAdapter, is there a upper limit for IPQPSRateLimiterAdapte?

Is there official benchmark performance data in documents? I can compare it and make sure whether it has space to optimize.

[m-o-m12]

@TonyS001 If testing the api through one IP, you should config IPQPSRateLimiterAdapte and QpsRateLimiterAdapter
strategy at the same time, because IPQPSRateLimiterAdapte strategy is to configure the maximum number of accesses per second for an IP on the interface. and the QpsRateLimiterAdapter strategy is to configure the maximum number of accesses per second on the interface. The default value of both of them are all 1000 qps.

[TonyS001]

@m-o-m12 Thanks a lot.
I set both two values 20000. The performance is better than before, reach 4k qps, but is still far below expectations.
At first the response time is pretty normal, but as test time grows, it gets very higher again.

Are there other relevant configure items that limit the performance? I attach the current config file, could you help check it, thanks!

main_net_config.txt

Is there official benchmark performance data? I am not sure whether 4k qps for json-rpc API is the peek performance.

[forfreeday]

@TonyS001
Benchmarks perform differently under different hardware conditions, you can show your hardware environment.

[PikaZ76]

@m-o-m12 Thanks a lot. I set both two values 20000. The performance is better than before, reach 4k qps, but is still far below expectations. At first the response time is pretty normal, but as test time grows, it gets very higher again.

Are there other relevant configure items that limit the performance? I attach the current config file, could you help check it, thanks!

main_net_config.txt

Is there official benchmark performance data? I am not sure whether 4k qps for json-rpc API is the peek performance.

any update on this bro?