synchronization is getting slower

[okmrs]

Hi dears.
I have sync problem, nearly like #685.
Following that issue, I tried almost everything introduced there.
However, it is not getting well, and then I need further support now.

The most serious problem I have is peer connection.
Even though modified max peer count, still less and poor connection (many of connecting nodes look like unhealthy).
I also tried to connect these nodes, but could not.
I guess that is due to busy connection.

Is there another solution or thing I should try?

[joslee7410]

Same here Sir, All my node getting very slow and slow. Already out of sync around 6000+- blocks.

[joslee7410]

I having 2 node, both nodes ara stucking at block 39600557 .

General
Num peers = 103
Sync mode = full

Current Header
Hash   = 0x691eaa3b4fe221c361337d1399ee3000de9edd1c5c3e1a7b589c213cd7fcc59e
Number = 39600557

Current Block
Hash   = 0x691eaa3b4fe221c361337d1399ee3000de9edd1c5c3e1a7b589c213cd7fcc59e
Number = 39600557

Syncing
Current block  = 39600557
Highest block  = 39608316
Starting block = 39600557

[sugeladi]

I having 2 node, both nodes ara stucking at block 39600557 .

General
Num peers = 103
Sync mode = full

Current Header
Hash   = 0x691eaa3b4fe221c361337d1399ee3000de9edd1c5c3e1a7b589c213cd7fcc59e
Number = 39600557

Current Block
Hash   = 0x691eaa3b4fe221c361337d1399ee3000de9edd1c5c3e1a7b589c213cd7fcc59e
Number = 39600557

Syncing
Current block  = 39600557
Highest block  = 39608316
Starting block = 39600557

I had the same problem. Have you solved it now?

[brucexiu]

I having 2 node, both nodes ara stucking at block 39600557 .

General
Num peers = 103
Sync mode = full

Current Header
Hash   = 0x691eaa3b4fe221c361337d1399ee3000de9edd1c5c3e1a7b589c213cd7fcc59e
Number = 39600557

Current Block
Hash   = 0x691eaa3b4fe221c361337d1399ee3000de9edd1c5c3e1a7b589c213cd7fcc59e
Number = 39600557

Syncing
Current block  = 39600557
Highest block  = 39608316
Starting block = 39600557

Polygon chain had a long reorg at the height 39599624

Please attach to your node and set a head manually.

[joslee7410]

I having 2 node, both nodes ara stucking at block 39600557 .

General
Num peers = 103
Sync mode = full

Current Header
Hash   = 0x691eaa3b4fe221c361337d1399ee3000de9edd1c5c3e1a7b589c213cd7fcc59e
Number = 39600557

Current Block
Hash   = 0x691eaa3b4fe221c361337d1399ee3000de9edd1c5c3e1a7b589c213cd7fcc59e
Number = 39600557

Syncing
Current block  = 39600557
Highest block  = 39608316
Starting block = 39600557

I had the same problem. Have you solved it now?

Not yet, and setHead seem not work as well.

[brucexiu]

I having 2 node, both nodes ara stucking at block 39600557 .

General
Num peers = 103
Sync mode = full

Current Header
Hash   = 0x691eaa3b4fe221c361337d1399ee3000de9edd1c5c3e1a7b589c213cd7fcc59e
Number = 39600557

Current Block
Hash   = 0x691eaa3b4fe221c361337d1399ee3000de9edd1c5c3e1a7b589c213cd7fcc59e
Number = 39600557

Syncing
Current block  = 39600557
Highest block  = 39608316
Starting block = 39600557

I had the same problem. Have you solved it now?

Not yet, and setHead seem not work as well.

It works for me. Start to sync, but still slow.

[joslee7410]

Num peers = 216
Sync mode = full

Current Header
Hash   = 0xf8e242cca1c2ac22aa2d0de7debf44984a9e2297730078acfa2cb74e4a2443af
Number = 39599664

Current Block
Hash   = 0xf8e242cca1c2ac22aa2d0de7debf44984a9e2297730078acfa2cb74e4a2443af
Number = 39599664

Syncing
Current block  = 39599664
Highest block  = 39611523
Starting block = 39599664

Now stuck at this block.

Because my bad block like this

 ########## BAD BLOCK #########
 Chain config: {ChainID: 137 Homestead: 0 DAO: <nil> DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 3395000, Muir Glacier: 3395000, Berlin: 14750000, London: 23850000, Arrow Glacier: <nil>, MergeFork: <nil>, Terminal TD: <nil>, Engine: bor}
 Number: 39599664
 Hash: 0xacaea68fe3c325b8790d493081e3c4fd477ff4b377aa90309b6cd55dd91484b3

[brucexiu]

39599664

Try to set the head as 39599000.

[sugeladi]

I have set head to 39599624 and am currently synchronizing blockchain. In addition, I would like to ask how you know that the reorganization took place at 39599624. Thank you very much @brucexiu

[brucexiu]

I have set head to 39599624 and am currently synchronizing blockchain. In addition, I would like to ask how you know that the reorganization took place at 39599624. Thank you very much @brucexiu

Please check this link https://polygonscan.com/blocks_forked

And I find this on Twitter.

[joslee7410]

I just tried with the height 39599624, works too.

Right now syncing. Hope will catched up soon. Thanks guys.

[okmrs]

I'm also running two nodes, but in my case, they don't stuck.
So still waiting for another support.

[okmrs]

Refer to this comment, next I should try upgrade to v0.3.4.

[joslee7410]

Refer to this comment, next I should try upgrade to v0.3.4.

Did upgrade to v0.3.4 solve the sync speed problem?

[okmrs]

Not upgraded yet. Now evaluating changelogs.

This issue affects the most major explorer...
https://cointelegraph.com/news/polygon-blockchain-reportedly-suffers-network-outage/amp

[sugeladi]

I have set head to 39599624 and am currently synchronizing blockchain. In addition, I would like to ask how you know that the reorganization took place at 39599624. Thank you very much @brucexiu

set head to 39599000， I've caught up with the latest block now. A total of 24 hours

[okmrs]

Upgrade to v0.3.4 is not effective, still sync slowly.

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[manav2401]

Hi @okmrs, apologies for the delay.

I need some more info into this issue. Can you help me with

1. The specs of the machine and config (either the whole config.toml or important params)
2. Some logs when the sync is slow.
3. If you see any messages like "Synchronisation failed" in the logs and if yes, can you post some of them here with full message?

[dougrchamberlain]

I'd like to look at the code related to these areas? I really want to learn more about, and contribute to, this project. I can try to duplicate the setup, on my fresh machine. I'd love to hear back. on this. I'm pretty good at performance and synching code issues.

[manav2401]

Hi @dougrchamberlain thanks for your interest in debugging this issue.

To answer your question, I am not sure if we've still narrowed this down to a specific part of the code as there are many moving parts.

1. Our primary check is to see the machine specs because the node runs too many background process when importing blocks. If the specs and disk i/o are low, it might be one of the reason of slow sync.
2. Apart from that we generally are checking out the connectivity of the nodes and some metrics like how many peers is the node connected to, what's the import frequency, what's the frequency of "Sync failed" error logs and any additional context around it if required. Even in this, there might be multiple code paths which are responsible to this.

Currently, we're speculating that in (2), there's one such importing case where most of the errors might occur. We're doing some modifications in that portion to check for improvements. Here's the PR for that if you're interested (#796).

[okmrs]

@manav2401
Thanks for your support.
I checked the processes running on my node, and found that other process seriously affected disk i/o.
When stopped that ps, bor sync got faster.

disk i/o are low, it might be one of the reason of slow sync.

This is very helpful for everyone facing this situation.
Even though machine spec(ex. disk size and cpu and memory) is enough, disk i/o may not be going well. It's blid spot.

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[manav2401]

Closing this for now, feel free to reopen.