Unable to sync full node, getting bad-witness-nonce-size on block height 434499

[dviator]

Hey all, running into an issue trying to perform a full sync of the chain.

On block height 434499 I see an error and the sync is unable to proceed. Restarting the node seems to have no effect.

Here's the error, which repeats ad nauseam.

[warning] (chain) Tried to connect invalid block: 0000000000000000010821c880d75e6e325705bf19f01f3a5a9399fa5d4f06e6 (434499).
[warning] (net) Error: Verification failure: bad-witness-nonce-size (code=invalid score=100 hash=0000000000000000010821c880d75e6e325705bf19f01f3a5a9399fa5d4f06e6)

I have run into this issue with 2 different bcoin processes now. One is running bcoin:v1.0.0-pre on node v9.11.2, the other is running bcoin:1.0.2 on node v10.16.2. These are running in docker containers inside a Docker Swarm. Both have the same config:

http-host: 0.0.0.0
index-address: true
index-tx: true
log-level: warning
network: main
no-auth: true
prefix: /data
selfish: true
max-outbound: 32

Wondering why this error is coming up, as this appears to be a valid block, and wondering what I can do to resolve the issue and get a full node synced? It seems others are able to complete full node syncs, so I'm also wondering what is so unique about our config that may be causing this issue.

Happy to provide any additional info needed.

[pinheadmz]

I think this error comes up when a legacy node sends us a block with no witness data. Score 100 means that peer is banned. So it’s strange that you would get this repeatedly (from multiple peers?) and especially strange on two different nodes. Can you tell how many peers you have? Is there anything else unusual about your configuration? Any other log messages before the invalid block? Set loglevel to ‘spam’ and see if anything else comes up.

[pinheadmz]

Ok there must be something extra weird happening here. This block is time stamped 10/15/2016, 4:21:32 PM PDT — segwit wasn’t activated yet! The witness check should not be active at this height either.

[dviator]

Right?! Yea we did notice that this block was from before segwit was enabled, should have mentioned in my initial post. But yes, it looks like this witness check is being triggered incorrectly for some reason.

Appears to be 32 peers, and this has been kept up pretty regularly. Anyway, since this cycle has been repeating for nearly 24 hours, it seems that no peer is able to send our nodes a block that they consider valid.

"pool": {
    "host": "3.92.81.167",
    "port": 8333,
    "agent": "/bcoin:1.0.2/",
    "services": "1000",
    "outbound": 32,
    "inbound": 0
  },

There's nothing else unusual about our configuration as far as I can tell.

Here's a log snippet with spam enabled. I didn't catch any additional info there but maybe you'll have better luck.

[debug] (peer) Requesting headers packet from peer with getheaders (75.40.139.200:8333).
[debug] (peer) Sending getheaders (hash=000000000000000001347da821b5ef2256cc1fc5a20e93d16070bd21332830b3, stop=0000000000000000038cc0f7bcdbb451ad34a458e2d535764f835fdeb896f29b).
[info] (net) Connected to 18.218.72.156:8333.
[info] (net) Received version (18.218.72.156:8333): version=70015 height=565799 services=10000001101 agent=/Satoshi:0.17.1/
[debug] (peer) Received verack (18.218.72.156:8333).
[debug] (peer) Version handshake complete (18.218.72.156:8333).
[info] (peer) Initializing witness compact blocks (18.218.72.156:8333).
[info] (peer) Peer initialized compact blocks (mode=0, version=2) (18.218.72.156:8333).
[debug] (peer) Peer sent a duplicate sendcmpct (18.218.72.156:8333).
[spam] (net) Received inv packet with 1 items: blocks=0 txs=1 (178.128.201.106:8333).
[debug] (net) Received 2000 headers from peer (75.40.139.200:8333).
[debug] (peer) Requesting headers packet from peer with getheaders (75.40.139.200:8333).
[debug] (peer) Sending getheaders (hash=000000000000000002980f6b89dbd3d75c99cbd9ba967ac85406f6d81e85b5c4, stop=0000000000000000038cc0f7bcdbb451ad34a458e2d535764f835fdeb896f29b).
[debug] (net) Received 2000 headers from peer (75.40.139.200:8333).
[debug] (peer) Requesting headers packet from peer with getheaders (75.40.139.200:8333).
[debug] (peer) Sending getheaders (hash=000000000000000003508fa79aeb5962e50b2c696911a572e3e5699733aebcdd, stop=0000000000000000038cc0f7bcdbb451ad34a458e2d535764f835fdeb896f29b).
[debug] (net) Received 1502 headers from peer (75.40.139.200:8333).
[debug] (net) Requesting 5502/5502 blocks from peer with getdata (75.40.139.200:8333).
[spam] (net) Received inv packet with 35 items: blocks=0 txs=35 (47.89.179.181:8333).
[spam] (net) Received inv packet with 35 items: blocks=0 txs=35 (62.212.83.41:8333).
[spam] (net) Received inv packet with 16 items: blocks=0 txs=16 (62.107.200.30:8333).
[spam] (net) Received inv packet with 1 items: blocks=0 txs=1 (178.128.201.106:8333).
[spam] (net) Received inv packet with 8 items: blocks=0 txs=8 (109.198.191.22:8333).
[spam] (net) Received inv packet with 1 items: blocks=0 txs=1 (178.128.201.106:8333).
[spam] (net) Received inv packet with 1 items: blocks=0 txs=1 (178.128.201.106:8333).
[warning] (chain) Tried to connect invalid block: 0000000000000000010821c880d75e6e325705bf19f01f3a5a9399fa5d4f06e6 (434499).
[debug] (peer) Rejecting block 0000000000000000010821c880d75e6e325705bf19f01f3a5a9399fa5d4f06e6 (75.40.139.200:8333): code=invalid reason=bad-witness-nonce-size.
[debug] (peer) Sending reject packet to peer (75.40.139.200:8333).
[debug] (peer) Ban threshold exceeded (75.40.139.200:8333).
[debug] (net) Banning peer (75.40.139.200:8333).
[info] (net) Removed loader peer (75.40.139.200:8333).
[info] (net) Initialized header chain to height 434498 (checkpoint=0000000000000000038cc0f7bcdbb451ad34a458e2d535764f835fdeb896f29b).
[warning] (net) Error: Verification failure: bad-witness-nonce-size (code=invalid score=100 hash=0000000000000000010821c880d75e6e325705bf19f01f3a5a9399fa5d4f06e6)

[braydonf]

One is running bcoin:v1.0.0-pre on node v9.11.2, the other is running bcoin:1.0.2 on node v10.16.2.

Where are you installing from, and are you using version 1.0.2 from npmjs.com, via git or the current master branch?

A few notes on the environment and installation:

• There is a different known issue with using Node.js v9.9.0 (see Memory exhaustion crash with full node sync (Node.js v9.9.0) #754), that will lead to memory exhaustion.
• The current LTS release is Node.js v12, and I think the most commonly used currently.
• It's recommended to install via git and verify integrity and signatures, as this isn't a supported feature of npm.
• Verify that you have dependencies installed for building native bindings, see https://github.com/bcoin-org/bcoin/blob/master/docs/getting-started.md#requirements.

[dviator]

Yes! Your comment nudged me in the right direction to find the issue. It turns out unbeknownst to me that my docker image was being built off of a fork of this code base, which had some unusual modifications in order to work around a long forgotten problem. (Someone else built the docker image some time ago.)

The changed lines were
prematureWitness: this.config.bool('premature-witness'),
in lib/node/fullnode.js and
VERIFY_WITNESS: 1, // 1
in lib/script/common.js

After spotting that and building off the proper codebase things started working right away, and my sync was able to progress past that block.

Thanks for the help guys!