Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate running lodestar on Nimbus's insecura - the "long ranged attacked" pyrmont fork #3658

Closed
5 tasks done
g11tech opened this issue Jan 22, 2022 · 9 comments
Closed
5 tasks done

Investigate running lodestar on Nimbus's insecura - the "long ranged attacked" pyrmont fork #3658

g11tech opened this issue Jan 22, 2022 · 9 comments
Assignees
@g11tech

Comments

@g11tech
Copy link
Contributor

g11tech commented Jan 22, 2022 •
edited
Loading

Nimbus team created a long range attack scenario on Pyrmont network as described here: https://notes.status.im/nimbus-insecura-network?view
tl-dr: They rewrote pyrnmont chain from epoch 60000 for 12k validators way back by using these validators (leaking the other 100k something validators on this alternate chain) so that this alternate chain finally finalized after exiting the majority "non attacking validators".
Now these node/validators are serving alternate chain. We want to test out the wss sync related issues using lodestar.

Check test following scenarios:

  • Scenario 1: sync on the attacking chain using a compromised checkpoint and see further finalization: see comment
  • Scenario 2: sync from the non attacked checkpoint (<= epoch 60k) to the attacking chain using the compromised bootnodes UPDATE: couldn't sync see comment
  • Scenario 3: syncing the prater on forward sync from blank db see comment
    • Sync started - OPINION : sync shouldn't have started
    • validate if it sync to the present on the attacking chain

UPDATE: following two scenarios can't be tested because insecura is (invalid) continuation of the abondoned pyrmont chain, so no checkpoint exists on pyrmont which is not part of insecura to test the following two points:
- [ ] Scenario 4: invalidate the sync on the attacking chain with compromised bootnodes but using a valid forward checkpoint (using the forward checkpoint PR) #3589
- [ ] Scenario 5: Invalidate the backfill sync on the attacking chain with a wss sync but a valid checkpoint in past from the normal chain
@g11tech g11tech self-assigned this Jan 22, 2022
@g11tech
Copy link
Contributor Author

g11tech commented Jan 22, 2022

UPDATE: Nimbus's wss endpoint seems to be loaded/stuck, not moving past fetching the state from their provided sync url :

bash-5.1# ./lodestar beacon --network pyrmont --eth1.enabled false --rootDir /data/insecura --weakSubjectivitySyncLatest --weakSubjectivityServerUrl http://insecura.nimbus.team
Jan-22 10:37:05.729 []                 info: Lodestar version=v0.26.0/master/+465/f9cb593d (git), network=pyrmont
Jan-22 10:37:05.741 [DB]               info: Connected to LevelDB database name=/data/insecura/chain-db
Jan-22 10:37:05.742 []                 info: Fetching weak subjectivity state from http://insecura.nimbus.team/eth/v2/debug/beacon/states/finalized

^CJan-22 10:37:41.851 []                 info: Stopping gracefully, use Ctrl+C again to force process exit
bash-5.1# ./lodestar beacon --network pyrmont --eth1.enabled false --rootDir /data/insecura --weakSubjectivitySyncLatest --weakSubjectivityServerUrl http://insecura.nimbus.team --network.discv5.bootEnrs enr:-LK4QGQl-6vfK7JdE8zPVzk9rMrXI1myBuy9xFZfA4JgEJm3ZScaoyyfPy3t6X57tY2G7gLK9zMzSkuFI1Hf0kRv7r4Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQNmNsAV4a5TWnfEArZrHUim553RymwmAbpAZxc3RoP1S4N0Y3CCIyiDdWRwgiMo enr:-LK4QKbOD5MlwM_BE7uVLwpRIwhlsFxiiOyReRhUBc7jgzblf8EXG5MccDxuRChfRqthkSo4wwC_ICECy2u8QVjx1esBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQOV-szjqex-IuG7NeNcIhp6jvc8j-hsFnNi14a9FTYGToN0Y3CCIyqDdWRwgiMq enr:-LK4QNymCG1DTf2rrVqzz9yTwU3-lE8_qrufLtUmlGU7nlsib4QNEpr_Csm_hfAjnjYL7uDcjTRkLLti9lwZH_UxnaQBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQMZ5KZLn-wqfc1FBWmeIn2lqsMmeUPI7nCerJQuBrOzLIN0Y3CCIymDdWRwgiMp
Jan-22 10:38:53.208 []                 info: Lodestar version=v0.26.0/master/+465/f9cb593d (git), network=pyrmont
Jan-22 10:38:53.219 [DB]               info: Connected to LevelDB database name=/data/insecura/chain-db
Jan-22 10:38:53.221 []                 info: Fetching weak subjectivity state from http://insecura.nimbus.team/eth/v2/debug/beacon/states/finalized

@g11tech
Copy link
Contributor Author

g11tech commented Jan 22, 2022 •
edited
Loading

UPDATE: synced on the attacking/alternate/invalid chain using compromised wss checkpoint and compromised bootnodes serving invalid chain

an-22 12:19:13.001 []                 info: Synced - slot: 3096095 - head: 3096095 0x04ee…2461 - finalized: 0xa3c0…d03e:96748 - peers: 2
Jan-22 12:19:25.001 []                 info: Synced - slot: 3096096 (skipped 1) - head: 3096095 0x04ee…2461 - finalized: 0x7994…ffb9:96749 - peers: 2
Jan-22 12:19:37.001 []                 info: Synced - slot: 3096097 - head: 3096097 0xc9a6…f93b - finalized: 0x7994…ffb9:96749 - peers: 2

Sucessfully backfilling from compromised checkpoint:
image

@g11tech
Copy link
Contributor Author

g11tech commented Jan 24, 2022

UPDATE: Error while trying to sync from a finalized checkpoint < epoch 60,000 (a correct checkpoint) but from the attacking chain as the checkpoint way behind the current clock epoch

bash-5.1# ./lodestar beacon --network pyrmont --metrics.enabled  --eth1.enabled false --rootDir /data/insecura1 --network.discv5.bootEnrs enr:-LK4QGQl-6vfK7JdE8zPVzk9rMrXI1myBuy9xFZfA4JgEJm3ZScaoyyfPy3t6X57tY2G7gLK9zMzSkuFI1Hf0kRv7r4Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQNmNsAV4a5TWnfEArZrHUim553RymwmAbpAZxc3RoP1S4N0Y3CCIyiDdWRwgiMo enr:-LK4QKbOD5MlwM_BE7uVLwpRIwhlsFxiiOyReRhUBc7jgzblf8EXG5MccDxuRChfRqthkSo4wwC_ICECy2u8QVjx1esBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQOV-szjqex-IuG7NeNcIhp6jvc8j-hsFnNi14a9FTYGToN0Y3CCIyqDdWRwgiMq enr:-LK4QNymCG1DTf2rrVqzz9yTwU3-lE8_qrufLtUmlGU7nlsib4QNEpr_Csm_hfAjnjYL7uDcjTRkLLti9lwZH_UxnaQBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQMZ5KZLn-wqfc1FBWmeIn2lqsMmeUPI7nCerJQuBrOzLIN0Y3CCIymDdWRwgiMp --weakSubjectivityServerUrl http://insecura.nimbus.team --weakSubjectivitySyncLatest --weakSubjectivityCheckpoint 0x1c8fc8f25001641b25a3572837962684def277fa8f525a86c349ccd8fd2d818b:59998
Jan-24 10:00:09.470 []                 info: Lodestar version=v0.26.0/g11tech/wssurlversion/+469/e6f2585a (git), network=pyrmont
Jan-24 10:00:09.612 [DB]               info: Connected to LevelDB database name=/data/insecura1/chain-db
Jan-24 10:00:10.834 []                 info: Fetching weak subjectivity state weakSubjectivityServerUrl=http://insecura.nimbus.team
 ✖ Error: Fetched weak subjectivity checkpoint not within weak subjectivity period.
    at initAndVerifyWeakSubjectivityState (/usr/app/packages/cli/src/cmds/beacon/initBeaconState.ts:58:11)
    at initBeaconState (/usr/app/packages/cli/src/cmds/beacon/initBeaconState.ts:115:12)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.beaconHandler [as handler] (/usr/app/packages/cli/src/cmds/beacon/handler.ts:75:41)

@g11tech
Copy link
Contributor Author

g11tech commented Jan 24, 2022 •
edited
Loading

Wss sync using a checkpoint from correct pyrmont chain but syncing via attacking chain failed because it seems that the pyrmont chain isn't finalizing, as it has been abandoned

curl https://************@eth2-beacon-prater.infura.io/eth/v1/beacon/states/finalized/finality_checkpoints 
{"data":{"previous_justified":{"epoch":"69034","root":"0x86144efe27f6e7338bd7480568b2d1b256fd0ee0757fe58978e82add875613e4"},"current_justified":{"epoch":"69035","root":"0xde0c88c56b246fbf90429992f565e9972d52305e3fe4e5da172951aee5910a79"},"finalized":{"epoch":"69034","root":"0x86144efe27f6e7338bd7480568b2d1b256fd0ee0757fe58978e82add875613e4"}}}

lodestar:

./lodestar beacon --network pyrmont --metrics.enabled  --eth1.enabled false --rootDir /data/insecura1 --network.discv5.bootEnrs enr:-LK4QGQl-6vfK7JdE8zPVzk9rMrXI1myBuy9xFZfA4JgEJm3ZScaoyyfPy3t6X57tY2G7gLK9zMzSkuFI1Hf0kRv7r4Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQNmNsAV4a5TWnfEArZrHUim553RymwmAbpAZxc3RoP1S4N0Y3CCIyiDdWRwgiMo enr:-LK4QKbOD5MlwM_BE7uVLwpRIwhlsFxiiOyReRhUBc7jgzblf8EXG5MccDxuRChfRqthkSo4wwC_ICECy2u8QVjx1esBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQOV-szjqex-IuG7NeNcIhp6jvc8j-hsFnNi14a9FTYGToN0Y3CCIyqDdWRwgiMq enr:-LK4QNymCG1DTf2rrVqzz9yTwU3-lE8_qrufLtUmlGU7nlsib4QNEpr_Csm_hfAjnjYL7uDcjTRkLLti9lwZH_UxnaQBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQMZ5KZLn-wqfc1FBWmeIn2lqsMmeUPI7nCerJQuBrOzLIN0Y3CCIymDdWRwgiMp --weakSubjectivityServerUrl http://insecura.nimbus.team --weakSubjectivitySyncLatest --weakSubjectivityCheckpoint 0x86144efe27f6e7338bd7480568b2d1b256fd0ee0757fe58978e82add875613e4:69034
Jan-24 10:11:53.198 []                 info: Lodestar version=v0.26.0/g11tech/wssurlversion/+469/e6f2585a (git), network=pyrmont
Jan-24 10:11:53.208 [DB]               info: Connected to LevelDB database name=/data/insecura1/chain-db
Jan-24 10:11:54.355 []                 info: Fetching weak subjectivity state weakSubjectivityServerUrl=http://insecura.nimbus.team
 ✖ Error: Unable to fetch weak subjectivity state: request to http://insecura.nimbus.team/eth/v2/debug/beacon/states/2209088 failed, reason: connect ECONNREFUSED 65.21.196.45:80
    at fetchWeakSubjectivityState (/usr/app/packages/cli/src/networks/index.ts:168:11)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at initBeaconState (/usr/app/packages/cli/src/cmds/beacon/initBeaconState.ts:109:37)
    at Object.beaconHandler [as handler] (/usr/app/packages/cli/src/cmds/beacon/handler.ts:75:41)

@g11tech
Copy link
Contributor Author

g11tech commented Jan 24, 2022 •
edited
Loading

lodestar started syncing on from blank db without using checkpoint sync...

./lodestar beacon --network pyrmont --metrics.enabled  --eth1.enabled false --rootDir /data/insecura --network.discv5.bootEnrs enr:-LK4QGQl-6vfK7JdE8zPVzk9rMrXI1myBuy9xFZfA4JgEJm3ZScaoyyfPy3t6X57tY2G7gLK9zMzSkuFI1Hf0kRv7r4Bh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQNmNsAV4a5TWnfEArZrHUim553RymwmAbpAZxc3RoP1S4N0Y3CCIyiDdWRwgiMo enr:-LK4QKbOD5MlwM_BE7uVLwpRIwhlsFxiiOyReRhUBc7jgzblf8EXG5MccDxuRChfRqthkSo4wwC_ICECy2u8QVjx1esBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQOV-szjqex-IuG7NeNcIhp6jvc8j-hsFnNi14a9FTYGToN0Y3CCIyqDdWRwgiMq enr:-LK4QNymCG1DTf2rrVqzz9yTwU3-lE8_qrufLtUmlGU7nlsib4QNEpr_Csm_hfAjnjYL7uDcjTRkLLti9lwZH_UxnaQBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpB0ZesLAQAgCf__________gmlkgnY0gmlwhEEVxC2Jc2VjcDI1NmsxoQMZ5KZLn-wqfc1FBWmeIn2lqsMmeUPI7nCerJQuBrOzLIN0Y3CCIymDdWRwgiMp
Jan-24 10:48:33.411 []                 info: Lodestar version=v0.26.0/g11tech/wssurlversion/+469/e6f2585a (git), network=pyrmont
Jan-24 10:48:33.419 [DB]               info: Connected to LevelDB database name=/data/insecura/chain-db
Jan-24 10:48:39.611 []                 info: Initializing beacon state from anchor state slot=0, epoch=0, stateRoot=0x2bb257ca66d05a047a65fe43a5f457b674de445d917cca029efb09b3ba4758c4
Jan-24 10:48:43.480 [SYNC]             info: BackfillSync - initializing from Checkpoint root=0x6a89af5df908893eedbed10ba4c13fc13d5653ce57db637e3bfded73a987bb87, epoch=0, slot=0
Jan-24 10:48:43.488 [METRICS]          info: Starting metrics HTTP server port=8008
(node:7245) MaxListenersExceededWarning: Possible EventEmitter memory leak detec

UPDATE: still syncing :

Jan-25 05:26:13.001 []                 info: Syncing - 2.6 days left - 10.3 slots/s - slot: 3115630 (skipped 2341060) - head: 774570 0x89d8…5f73 - finalized: 0x316b…f0fc:24203 - peers: 8

###############################

UPDATE: sync stalled near epoch at which pyrmont was abandoned (~60,000)

Jan-26 15:22:49.002 []                 info: Searching peers - peers: 0 - slot: 3125813 (skipped 1082143) - head: 2043670 0x5275…15b2 - finalized: 0x1c8f…818b:59998
Jan-26 15:23:01.000 []                 info: Searching peers - peers: 0 - slot: 3125814 (skipped 1082144) - head: 2043670 0x5275…15b2 - finalized: 0x1c8f…818b:59998
Jan-26 15:23:13.000 []                 info: Searching peers - peers: 0 - slot: 3125815 (skipped 1082145) - head: 2043670 0x5275…15b2 - finalized: 0x1c8f…818b:59998

However on cleaning peer and restarting, it started syncing syncing on the insecura chain :

an-26 15:26:01.000 []                 info: Searching peers - peers: 0 - slot: 3125829 (skipped 1206149) - head: 1919680 0xc538…cfcd - finalized: 0xf09c…7ece:59988
Jan-26 15:26:13.007 []                 info: Syncing - ? left - 0.00 slots/s - slot: 3125830 (skipped 1206150) - head: 1919680 0xc538…cfcd - finalized: 0xf09c…7ece:59988 - peers: 3
Jan-26 15:26:25.001 []                 info: Syncing - 4.5 days left - 3.12 slots/s - slot: 3125831 (skipped 1205923) - head: 1919908 0x05c2…e0b7 - finalized: 0x5030…c8f0:59995 - peers: 3
Jan-26 15:26:37.033 []                 info: Syncing - 2.5 days left - 5.59 slots/s - slot: 3125832 (skipped 1205738) - head: 1920094 0x4171…6d32 - finalized: 0x2ccd…57e9:60001 - peers: 4
Jan-26 15:26:49.000 []                 info: Syncing - 1.9 days left - 7.42 slots/s - slot: 3125833 (skipped 1205557) - head: 1920276 0x3210…cf05 - finalized: 0xbeb0…f279:60006 - peers: 4
Jan-26 15:27:01.001 []                 info: Syncing - 1.6 days left - 8.73 slots/s - slot: 3125834 (skipped 1205385) - head: 1920449 0xae27…8be4 - finalized: 0x71a8…9f50:60012 - peers: 4

UPDATE: sucessfull synced to the insecura chainhead as expected

an-27 18:17:37.001 []                 info: Synced - slot: 3133887 (skipped 1) - head: 3133886 0x09ff…b8a1 - finalized: 0x6869…baf1:96544 - peers: 23
Jan-27 18:17:49.001 []                 info: Synced - slot: 3133888 (skipped 2) - head: 3133886 0x09ff…b8a1 - finalized: 0x6869…baf1:96544 - peers: 23
Jan-27 18:18:01.001 []                 info: Synced - slot: 3133889 - head: 3133889 0x26f3…559d - finalized: 0x6869…baf1:96544 - peers: 23
Jan-27 18:18:13.001 []                 info: Synced - slot: 3133890 (skipped 1) - head: 3133889 0x26f3…559d - finalized: 0x6869…baf1:96544 - peers: 23
Jan-27 18:18:25.000 []                 info: Synced - slot: 3133891 (skipped 2) - head: 3133889 0x26f3…559d - finalized: 0x6869…baf1:96544 - peers: 23

@g11tech
Copy link
Contributor Author

g11tech commented Feb 3, 2022

CLOSING issue as Scenario 4,5 not possible to check, see description for the reasons.

@g11tech g11tech closed this as completed Feb 3, 2022
@dapplion
Copy link
Contributor

dapplion commented Feb 4, 2022

Thank you so much for doing this tests! Love to see the security protections in action

@g11tech
Copy link
Contributor Author

g11tech commented Feb 4, 2022

Thank you so much for doing this tests! Love to see the security protections in action

@dapplion Should we apply weak subjectivity timeperiod checks on start from blank DB as well (Scenario 3 in the description)? which we don't do right now. This means one would not be able to start from a blank DB without providing a checkpoint.

@dapplion
Copy link
Contributor

dapplion commented Feb 4, 2022 •
edited
Loading

Should we apply weak subjectivity timeperiod checks on start from blank DB as well (Scenario 3 in the description)? which we don't do right now. This means one would not be able to start from a blank DB without providing a checkpoint.

What do other clients do? In that case you would do the WS math on the genesis state?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@g11tech g11tech

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dapplion @g11tech