bps slow and cannot catch up the latest block

[zhangxf55]

Describe the bug
Low bps while downloading blocks

To Reproduce

1. build from the source

git clone https://github.com/near/nearcore && \
cd nearcore && \
git fetch origin --tags
# 1.40
git checkout 7dd0b5993577f592be15eb102e5a3da37be66271
cd ~/nearcore && make release
mkdir -p /root/near/{bin,conf} /var/log/near /data/near
mv ~/nearcore/target/release/neard ~/near/bin/

2. init the genesis data and state

~/near/bin/neard --home /data/near init --chain-id mainnet --download-genesis --download-config
rm -rf /data/near/config.json
wget -O /data/near/config.json https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/mainnet/config.json

3. download the snapshot

cd /data/near && aws s3 --no-sign-request cp s3://near-protocol-public/backups/mainnet/rpc/latest . && \
LATEST=$(cat latest) && \
aws s3 --no-sign-request cp --no-sign-request --recursive s3://near-protocol-public/backups/mainnet/rpc/$LATEST /data/near/data

4. run the node

/root/near/bin/neard --home /data/near run

Expected behavior
The bps will decrease from about 2.0 to 0.5, the left number of blocks will increase, and the node will never keep up the latest block chain.

1. the bandwidth and the network status:

                                        19.1Mb                                  38.1Mb                                   57.2Mb                                  76.3Mb                             95.4Mb
└───────────────────────────────────────┴───────────────────────────────────────┴────────────────────────────────────────┴───────────────────────────────────────┴────────────────────────────────────────
near                                                                                    => 170-245-144-216.static.reverse.lstn.net                                                 1.51Mb  1.17Mb  1.17Mb
                                                                                        <=                                                                                         1.64Mb  1.22Mb  1.22Mb
near                                                                                    => 15.235.54.185                                                                           1.50Mb  1.17Mb  1.17Mb
                                                                                        <=                                                                                         1.64Mb  1.22Mb  1.22Mb
near                                                                                    => 69.67.151.93                                                                            1.50Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                         1.64Mb  1.22Mb  1.22Mb
near                                                                                    => 142-245-144-216.static.reverse.lstn.net                                                 1.51Mb  1.17Mb  1.17Mb
                                                                                        <=                                                                                         1.57Mb  1.21Mb  1.21Mb
near                                                                                    => 173.231.40.82                                                                           1.50Mb  1.17Mb  1.17Mb
                                                                                        <=                                                                                         1.58Mb  1.22Mb  1.22Mb
near                                                                                    => 34.89.217.176                                                                           1.51Mb  1.17Mb  1.17Mb
                                                                                        <=                                                                                         1.56Mb  1.21Mb  1.21Mb
near                                                                                    => 194.69.165.150                                                                          1.50Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                         1.56Mb  1.21Mb  1.21Mb
near                                                                                    => 178.162.169.185                                                                         1.51Mb  1.17Mb  1.17Mb
                                                                                        <=                                                                                         1.52Mb  1.19Mb  1.19Mb
near                                                                                    => static.47.90.130.94.clients.your-server.de                                              1.51Mb  1.17Mb  1.17Mb
                                                                                        <=                                                                                         1.48Mb  1.12Mb  1.12Mb
near                                                                                    => ns1020698.ip-15-204-64.us                                                               1.50Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                         1.58Mb   991Kb   991Kb
near                                                                                    => static.185.129.181.135.clients.your-server.de                                           1.51Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                         1.57Mb   968Kb   968Kb
near                                                                                    => static.150.216.181.135.clients.your-server.de                                           1.50Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                         1.22Mb   967Kb   967Kb
near                                                                                    => static.244.153.9.5.clients.your-server.de                                               1.50Mb  1.17Mb  1.17Mb
                                                                                        <=                                                                                         1.03Mb   964Kb   964Kb
near                                                                                    => ns3224190.ip-57-128-33.eu                                                               1.49Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                          859Kb   925Kb   925Kb
near                                                                                    => 142.132.132.227                                                                         1.50Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                          938Kb   883Kb   883Kb
near                                                                                    => 65.109.126.219                                                                          1.50Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                         1.39Mb   877Kb   877Kb
near                                                                                    => 216.144.245.122                                                                         1.49Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                          866Kb   876Kb   876Kb
near                                                                                    => 173.234.17.196                                                                          1.50Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                          950Kb   876Kb   876Kb
near                                                                                    => 65.109.67.50                                                                            1.49Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                          870Kb   875Kb   875Kb
near                                                                                    => 148.72.170.242                                                                          1.50Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                         1.37Mb   874Kb   874Kb
near                                                                                    => 34.135.231.166                                                                          1.50Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                          868Kb   868Kb   868Kb
near                                                                                    => 80.239.221.223                                                                          1.50Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                         1.10Mb   872Kb   872Kb
near                                                                                    => static.87.70.88.23.clients.your-server.de                                               1.50Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                          852Kb   865Kb   865Kb
near                                                                                    => 43.206.134.184                                                                          1.49Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                          825Kb   839Kb   839Kb
near                                                                                    => 44.201.205.167                                                                          1.49Mb  1.16Mb  1.16Mb
                                                                                        <=                                                                                          876Kb   837Kb   837Kb

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
TX:             cum:   18.6MB   peak:   48.0Mb                                                                                                                            rates:   48.0Mb  37.3Mb  37.3Mb
RX:                    14.5MB           35.4Mb                                                                                                                                     35.4Mb  29.1Mb  29.1Mb
TOTAL:                 33.2MB           83.4Mb                                                                                                                                     83.4Mb  66.4Mb  66.4Mb

2. the nvme disk io status:

root@near:~/near/bin# iostat /dev/nvme0n1
Linux 5.15.0-113-generic (near)         07/04/2024      _x86_64_        (32 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.40    0.00    1.28    2.85    0.00   92.46

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
nvme0n1        1304.50     65251.60     70864.11      5300.73 24890456669 27031369140 2021983556

3. the cpu and ram status:

4. the standard output of neard process:

5. DNAT configuration:

pkts bytes target     prot opt in     out     source               destination
  208 12352 DNAT       tcp  --  *      *       0.0.0.0/0            118.143.*.*      tcp dpt:24567 to:192.168.1.20:24567
14731  884K DNAT       tcp  --  *      *       0.0.0.0/0            118.143.*.*      tcp dpt:3030 to:192.168.1.20:3030

p.s. is the 3030 port forwarding necessary?

Version:

• nearcore: 1.40-commit-7dd0b5993577f592be15eb102e5a3da37be66271
• rust version: rustc 1.79.0 (129f3b996 2024-06-10)
• mainnet
• os: ubuntu 22.04
• network environment:

root@near:~/near/bin# curl https://ipinfo.io
{
  "ip": "118.143.*.*",
  "city": "Hong Kong",
  "region": "Hong Kong",
  "country": "HK",
  "loc": "22.2783,114.1747",
  "org": "AS9304 HGC Global Communications Limited",
  "timezone": "Asia/Hong_Kong",
  "readme": "https://ipinfo.io/missingauth"
}

[telezhnaya]

Hey!
Sorry for the late response. Is it still a relevant issue?