[2.6][T3] High incidence of forks on devnet (due to postBlock throttling?)

[alessiodf]

There has been a higher incidence of forks on devnet recently, all of which have been caused by delayed blocks on the network (i.e. delegate A's block is received during delegate B's slot, so both A and B forge at the same height and some parts of the network receive A's block first and disregard B's, whereas other parts receive B's block first so disregard A's).

It looks like the outgoing throttling of postBlock may be to blame as it will keep delaying the propagation by 1 second until the hasExceededRateLimit check is false. Even if it means the block's slot has passed.

It might just be a coincidence, but my logs do show quite a lot of Throttling outgoing requests to x.x.x.x/p2p.peer.postBlock to avoid triggering their rate limit messages and, empirically, there are quite a lot of forks on devnet which are caused by blocks arriving late and encroaching on the next delegate's slot.

[vasild]

Is something broken with the outgoing throttler or maybe I miss something - in order to trigger it one would need to send many blocks per second, however, there should be just 1 new block every 8 seconds. How could a throttler kick in with 1 block / 8 sec?

[alessiodf]

I agree with your logic @vasild so maybe something is amiss with the throttler because it's definitely throttling on postBlock occasionally but I don't know if that is the reason for the delayed blocks which are causing devnet forks.

[vasild]

While it remains unclear whether the throttle delay has been the reason for the forks, it was too restrictive - allowing just 1 block per 4 seconds, whereas with late blocks it may happen that we have two blocks in less than 4 seconds. That has been relaxed now to allow 2 blocks per 4 seconds.

So we should never see the Throttling outgoing requests to x.x.x.x/p2p.peer.postBlock to avoid triggering their rate limit messages again.

[alessiodf]

Some extra info requested by @supaiku0:

The nodes go through periods of receiving no blocks whatsoever (despite having sufficient peers). Then when a minute of inactivity elapses or during wakeup, it downloads the missing blocks. During this time if you're due to forge you're likely to either miss your slot or you may forge at the wrong height which is then accepted by other nodes that also didn't receive any blocks -- this may be what is causing the forking.

Additionally I tried disabling the outgoing rate limit throttler entirely on my node by making the method a no-op (and forcing it to peer with another node under my control to monitor it for received blocks) but it made no difference at all with the above sequence of events still occurring.

However, since the release of the next.7 version, I haven't seen any more Throttling outgoing requests to x.x.x.x/p2p.peer.postBlock to avoid triggering their rate limit messages as @vasild rightly said would be the case.

I hope this helps.

[alessiodf]

One other thing that came to mind is the limit for the number of peers per subnet/CIDR, in case devnet is too centralised around the same providers which is somehow causing a problem, but if that were the case I would have expected it to manifest itself in Core 2.5 when that was on devnet too, but IIRC that didn't happen.

[alessiodf]

This might be responsible for it in case for whatever reason the receiving node doesn't have the sending node in its peer list.

core/packages/core-p2p/src/socket-server/worker.ts

Lines 190 to 200 in bc71013

	if (["postTransactions", "postBlock"].includes(handler)) {
	// has to be in the peer list to use the endpoint
	const {
	data: { isPeerOrForger },
	} = await this.sendToMasterAsync("p2p.internal.isPeerOrForger", {
	data: { ip: req.socket.remoteAddress },
	});
	if (!isPeerOrForger) {
	req.socket.terminate();
	return;
	}

[air1one]

So now with the update of rate limiter that should not cause "postBlock throttling" anymore + disabling the peer check for postBlock and postTransaction that was done recently, I think we can close this.

Did you see recently those forks happening again @alessiodf or have any other reason we should keep this open ? Thanks in advance.