lotus-miner proving compute windowpost during lotus-miner storage cleanup crashes lotus-miner

[hyunmoon]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus fvm/fevm - Lotus FVM and FEVM interactions
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt/WinningPoSt)
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

Daemon:  1.21.0-rc1+mainnet+git.3f8a41cfe+api1.5.0
Local: lotus-miner version 1.21.0-rc1+mainnet+git.3f8a41cfe

Repro Steps

Run lotus-miner storage cleanup
Before cleanup is finished, run
lotus-miner proving compute window-post <partition_number>

Describe the Bug

lotus-miner proving compute windowpost during lotus-miner storage cleanup crashes lotus-miner.

lotus-miner proving compute window-post 12
Took 2.167362687s
ERROR: RPCConnectionError

Logging Information

-

[rjan90]

When you say crashes here, do you mean that the RPCConnection dies, or that the lotus-miner process crashes. Its unclear from the logging here.

Do youy have any logs from the time this was happening? In the meantime we will try to reproduce this error

[hyunmoon]

Both cleanup and proving compute ended with ERROR: RPCConnectionError and lotus-miner process crashed at the same time.

[hyunmoon]

I could not find anything special before the process crashed, so I didn't back up the log unfortunately.

[rjan90]

Can you check if there where any reports in dmesg/systemd?

[hyunmoon]

Today, lotus-miner crashed again despite I downgraded everything (node, miner, worker) back to v1.20.4

No dmesg logs at the time.
This time I backed up the log.

2023-03-29T14:47:02.515+0900	WARN	fsutil	fsutil/dealloc_linux.go:22	could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.516+0900	WARN	fsutil	fsutil/dealloc_linux.go:22	could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.520+0900	WARN	fsutil	fsutil/dealloc_linux.go:22	could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.521+0900	WARN	fsutil	fsutil/dealloc_linux.go:22	could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.522+0900	WARN	fsutil	fsutil/dealloc_linux.go:22	could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.524+0900	WARN	fsutil	fsutil/dealloc_linux.go:22	could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.526+0900	WARN	fsutil	fsutil/dealloc_linux.go:22	could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.528+0900	WARN	fsutil	fsutil/dealloc_linux.go:22	could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.529+0900	WARN	fsutil	fsutil/dealloc_linux.go:22	could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:02.529+0900	WARN	fsutil	fsutil/dealloc_linux.go:22	could not deallocate space, ignoring: operation not supported
2023-03-29T14:47:04.010+0900	WARN	sectors	pipeline/fsm.go:799	sector 325680 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325680},"ProofType":8},"aPQfsubT3uUQF3k3wZEzr8gK7r0X8hVpa4a9EREROf0=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqfd5ml5tztcfcydskrstmjhyn55bby6xb3ya27chssnadtdhfxefy"}}]]]), got ()
2023-03-29T14:47:04.012+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(325680), waiting 59.987845509s before retrying
2023-03-29T14:47:07.013+0900	WARN	sectors	pipeline/fsm.go:799	sector 326271 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326271},"ProofType":8},"uSVj9leeFVVrj2FwsiVPdeX43wx7Kir/inoqPKDzn+U=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqjhd4qyf3rpbnvpvlpg4iilo5dhbke23k7vw7jvxz2dzc37jld2mi"}}]]]), got ()
2023-03-29T14:47:07.015+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(326271), waiting 59.984417492s before retrying
2023-03-29T14:47:10.026+0900	INFO	miner	miner/miner.go:478	completed mineOne	{"tookMilliseconds": 22, "forRound": 2725415, "baseEpoch": 2725414, "baseDeltaSeconds": 10, "nullRounds": 0, "lateStart": false, "beaconEpoch": 2821260, "lookbackEpochs": 900, "networkPowerAtLookback": "22416530710540058624", "minerPowerAtLookback": "19488925475143680", "isEligible": true, "isWinner": false, "error": null}
2023-03-29T14:47:13.006+0900	WARN	sectors	pipeline/fsm.go:799	sector 325994 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325994},"ProofType":8},"ccJjop/rEPT9trRMBsUpaqRjSRvWNmcIC6Y2IKe5wpQ=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqglwaz47wfphh6swm53fk6w6235a66aokaudfiac3jng4aza2toea"}}]]]), got ()
2023-03-29T14:47:13.007+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(325994), waiting 59.992171923s before retrying
2023-03-29T14:47:16.008+0900	WARN	sectors	pipeline/fsm.go:799	sector 326190 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326190},"ProofType":8},"2Gxdy7CA3UujfxzdaATXW/bMgZ5rmcqKZ8icX27SvgE=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqnxeqn7ov4pmzp42fci4av6fqxpcm5aue2mymtwclgbv5mq2h6ini"}}]]]), got ()
2023-03-29T14:47:16.009+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(326190), waiting 59.990161952s before retrying
2023-03-29T14:47:17.006+0900	WARN	sectors	pipeline/fsm.go:799	sector 326248 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326248},"ProofType":8},"e4ZFQO51z8ggVz3LH/kCWX8qVaUDm2qI+x/hoM6NI5A=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqc67vn74jjp4sgshce4excjqbiptjdnrohtanvpoxvc43p2kdqily"}}]]]), got ()
2023-03-29T14:47:17.007+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(326248), waiting 59.992732461s before retrying
2023-03-29T14:47:22.024+0900	INFO	sectors	pipeline/states_sealing.go:571	scheduling seal proof computation...
2023-03-29T14:47:22.024+0900	INFO	sectors	pipeline/states_sealing.go:573	KOMIT 325933 7a56f40e26f5253a658d955e92639479537f3e85c3cb512d83ab8ff294c153bb(2723429); c04af2053f3f54cd32953bd132b28d59584a51dc4733351307bab36bd2e5909c(2724905); [{34359738368 baga6ea4seaqcaezwci2ryjn5jym4q5t2bhrkpkh2jbaar5spnoxr6tqi4zewyki}]; r:bagboea4b5abca3m4tz6655qppg6seupgiva6uanngwp7zd5tixmwp5i6m23lvrtn; d:baga6ea4seaqcaezwci2ryjn5jym4q5t2bhrkpkh2jbaar5spnoxr6tqi4zewyki
2023-03-29T14:47:22.025+0900	INFO	sectors	pipeline/states_failed.go:30	ComputeProofFailed(325933), waiting 59.974355867s before retrying
2023-03-29T14:47:27.013+0900	WARN	sectors	pipeline/fsm.go:799	sector 325572 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325572},"ProofType":8},"/SaZ/KkIdRPjYwOkAKuHuHyUp96bcJtjR01qKTSa3vc=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqhrydfzpnxugiv5of5dd6ylllp6kmf3lw5h7vueo7drz5frnz6kji"}}]]]), got ()
2023-03-29T14:47:27.015+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(325572), waiting 59.984434359s before retrying
2023-03-29T14:47:29.000+0900	WARN	sectors	pipeline/fsm.go:799	sector 326188 got error event sealing.SectorSealPreCommit1Failed: consecutive compute fails
2023-03-29T14:47:29.002+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(326188), waiting 59.997433153s before retrying
2023-03-29T14:47:29.004+0900	WARN	sectors	pipeline/fsm.go:799	sector 326431 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326431},"ProofType":8},"ZLZDGrNiFrzaV51oJbYWEl3mgjy+E6JnQVNkvA8TQ4Q=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqjanafdzq5dgncftaczypvitfpfrdyg25nku5xtql56efvq6756ba"}}]]]), got ()
2023-03-29T14:47:29.006+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(326431), waiting 59.993972217s before retrying
2023-03-29T14:47:29.009+0900	WARN	sectors	pipeline/fsm.go:799	sector 325682 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325682},"ProofType":8},"rQUxKgBvw2DirCnRFgY8Q1Cj87WgODDVU5kurPFfVjs=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqo34t2pne67riyxygptekaxkwiukhkvfgxo4e2m3ofy6khzdenukq"}}]]]), got ()
2023-03-29T14:47:29.011+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(325682), waiting 59.988613003s before retrying
2023-03-29T14:47:36.003+0900	WARN	sectors	pipeline/fsm.go:799	sector 326412 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326412},"ProofType":8},"SasEpJ5hrKQromSPyslCOCJvyjhG9aTCgwkEZ1iiLY4=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqmqewykpi6kpcbxyodo474rfghjxkprbn6zv3by3agb3isdudmkdi"}}]]]), got ()
2023-03-29T14:47:36.004+0900	WARN	sectors	pipeline/fsm.go:799	sector 326419 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326419},"ProofType":8},"Ic2uB56knTbkL9FAekHMIFzAxxAw5qYCTdZdhy6ncVE=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqcsxacgnuwu5ydq4vv4ebza4ffj7dphf2y6xy4xxjryejkcgrmkda"}}]]]), got ()
2023-03-29T14:47:36.004+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(326412), waiting 59.995356581s before retrying
2023-03-29T14:47:36.005+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(326419), waiting 59.994806411s before retrying
2023-03-29T14:47:36.006+0900	WARN	sectors	pipeline/fsm.go:799	sector 325989 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325989},"ProofType":8},"cDCNyshjkQZdVi0ujfyT86YGe9cBNbRwGRcPdS+wPWM=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqiqfjgz5df6xhjmbfbgbwz2nirfjomwc4todycgouhfgcxvno5eci"}}]]]), got ()
2023-03-29T14:47:36.008+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(325989), waiting 59.991381547s before retrying
2023-03-29T14:47:38.008+0900	WARN	sectors	pipeline/fsm.go:799	sector 325044 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325044},"ProofType":8},"3BDy6QSMyuKo/UlE1zeHh9j6Bh07AtkPaM2sya7z3Ck=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqnligaakkuxm2x3ez7cdvgkwbpqh4bpyvttplwie6bb6aqrngt6iq"}}]]]), got ()
2023-03-29T14:47:38.011+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(325044), waiting 59.988537095s before retrying
2023-03-29T14:47:39.007+0900	WARN	sectors	pipeline/fsm.go:799	sector 326265 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326265},"ProofType":8},"HDpxqyEgT/LvrBqlEoomDWmPe8c0jCrTu2lyqYMQwPQ=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqkcd4hrlhmsj2aruasausi3bf43uzligxekrrxvbgwpwgexydnema"}}]]]), got ()
2023-03-29T14:47:39.008+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(326265), waiting 59.991422116s before retrying
2023-03-29T14:47:39.011+0900	WARN	sectors	pipeline/fsm.go:799	sector 325432 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325432},"ProofType":8},"55U2MxORM9qHb3orIKdcZ3X4IKZHvjBkOWNQtZvi1LU=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqgezndmeopjqytlkf25rzmiwwv4gx5lb4sg7jjdxoag4lwkimvgiy"}}]]]), got ()
2023-03-29T14:47:39.016+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(325432), waiting 59.983983626s before retrying
2023-03-29T14:47:40.034+0900	INFO	miner	miner/miner.go:478	completed mineOne	{"tookMilliseconds": 15, "forRound": 2725416, "baseEpoch": 2725415, "baseDeltaSeconds": 10, "nullRounds": 0, "lateStart": false, "beaconEpoch": 2821261, "lookbackEpochs": 900, "networkPowerAtLookback": "22416531296104349696", "minerPowerAtLookback": "19488925475143680", "isEligible": true, "isWinner": false, "error": null}
2023-03-29T14:47:43.009+0900	WARN	sectors	pipeline/fsm.go:799	sector 325495 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325495},"ProofType":8},"D9lJoOO/wJPOG9CDHhWTOdaC2DSRx6Xt9aKf/LslyCs=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqloila7ojjsrc3anam53ypgutavuguhoik66cqgjhsic66wovc2ma"}}]]]), got ()
2023-03-29T14:47:43.011+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(325495), waiting 59.988686518s before retrying
2023-03-29T14:47:45.175+0900	WARN	sectors	pipeline/fsm.go:799	sector 326427 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: storage call error 0: cleaning up sealed data: remove from local: finding existing sector {1697248 326427}(t:2) failed: RPC client error: sendRequest failed: Post "http://10.10.35.7:2345/rpc/v0": read tcp 10.10.35.103:47112->10.10.35.7:2345: read: connection reset by peer [name: porep-35-103]: cleaning up sealed data: remove from local: finding existing sector {1697248 326427}(t:2) failed: RPC client error: sendRequest failed: Post "http://10.10.35.7:2345/rpc/v0": read tcp 10.10.35.103:47112->10.10.35.7:2345: read: connection reset by peer
2023-03-29T14:47:45.175+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(326427), waiting 59.824083523s before retrying
2023-03-29T14:47:45.888+0900	INFO	sectors	pipeline/states_sealing.go:429	submitting precommit for sector 326266 (deposit: 79102111037133547): 
2023-03-29T14:47:46.286+0900	INFO	sectors	pipeline/states_sealing.go:473	Sector precommitted: 326266
2023-03-29T14:47:51.009+0900	WARN	sectors	pipeline/fsm.go:799	sector 325871 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325871},"ProofType":8},"c+gAh98oVXKiGw6S/fHbAYF7q9SuiRvDdljqJSKig9M=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqkovovg2tipaofftwxtzylklewavbsxch42gmacuw2ayl6w2ux6ay"}}]]]), got ()
2023-03-29T14:47:51.011+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(325871), waiting 59.988097483s before retrying
2023-03-29T14:47:53.006+0900	WARN	sectors	pipeline/fsm.go:799	sector 325739 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325739},"ProofType":8},"7mPubSGG6Py2Jkk6/WP6IRTdJxmhW6BqUmuE49VhctQ=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqkfndsdaqmermbpexqslmvrxlfjrsrxo5ftjritcvdncxwaefokmq"}}]]]), got ()
2023-03-29T14:47:53.008+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(325739), waiting 59.991336617s before retrying
2023-03-29T14:47:59.726+0900	INFO	sectors	pipeline/states_sealing.go:571	scheduling seal proof computation...
2023-03-29T14:47:59.726+0900	INFO	sectors	pipeline/states_sealing.go:573	KOMIT 326208 8389cd049a29d71f0c81431998ef3d1e37c52152c541955dcde44cb8fc155b88(2723959); ffab43c1fa8aa41a4bb92442b4960cc725ffcc17c43d63eda1c6ef49aaf5bd52(2725410); [{34359738368 baga6ea4seaqbazty6cu2vfsew3u4xa25npfkbsybaf5mfvocgsgmfb5leptggoq}]; r:bagboea4b5abca57al6ar7dztfh67oslewjeuwsvcojpikuymswdjox52gwrir3bz; d:baga6ea4seaqbazty6cu2vfsew3u4xa25npfkbsybaf5mfvocgsgmfb5leptggoq
2023-03-29T14:47:59.727+0900	INFO	sectors	pipeline/states_sealing.go:571	scheduling seal proof computation...
2023-03-29T14:47:59.727+0900	INFO	sectors	pipeline/states_sealing.go:573	KOMIT 326129 10b65cc8b6fc694762373f9310653997c60b023ef588dc117c711a667c0027ff(2723786); ffab43c1fa8aa41a4bb92442b4960cc725ffcc17c43d63eda1c6ef49aaf5bd52(2725410); [{34359738368 baga6ea4seaqovryqvjzrofyw754ujfeaq3ouw2xxfnamhm3p3tiy6fmv356d6pi}]; r:bagboea4b5abcbl6ak3blmvs6er4jqc6z55t55swd5uzobtldne2g3bjxwskpmotq; d:baga6ea4seaqovryqvjzrofyw754ujfeaq3ouw2xxfnamhm3p3tiy6fmv356d6pi
2023-03-29T14:48:04.007+0900	WARN	sectors	pipeline/fsm.go:799	sector 325680 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325680},"ProofType":8},"aPQfsubT3uUQF3k3wZEzr8gK7r0X8hVpa4a9EREROf0=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqfd5ml5tztcfcydskrstmjhyn55bby6xb3ya27chssnadtdhfxefy"}}]]]), got ()
2023-03-29T14:48:04.009+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(325680), waiting 59.990061112s before retrying
2023-03-29T14:48:07.008+0900	WARN	sectors	pipeline/fsm.go:799	sector 326271 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":326271},"ProofType":8},"uSVj9leeFVVrj2FwsiVPdeX43wx7Kir/inoqPKDzn+U=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqjhd4qyf3rpbnvpvlpg4iilo5dhbke23k7vw7jvxz2dzc37jld2mi"}}]]]), got ()
2023-03-29T14:48:07.010+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(326271), waiting 59.989069806s before retrying
2023-03-29T14:48:10.018+0900	INFO	miner	miner/miner.go:478	completed mineOne	{"tookMilliseconds": 12, "forRound": 2725417, "baseEpoch": 2725416, "baseDeltaSeconds": 10, "nullRounds": 0, "lateStart": false, "beaconEpoch": 2821262, "lookbackEpochs": 900, "networkPowerAtLookback": "22416462473548398592", "minerPowerAtLookback": "19488925475143680", "isEligible": true, "isWinner": false, "error": null}
2023-03-29T14:48:12.491+0900	INFO	wdpost	wdpost/wdpost_run.go:412	computing window post	{"cycle": "2023-03-29T14:36:00.320+0900", "batch": 0, "elapsed": 263.640970975, "skip": 1, "err": "some sectors (1) were skipped", "errVerbose": "some sectors (1) were skipped:\n    github.com/filecoin-project/lotus/storage/sealer.(*Manager).generateWindowPoSt\n        /home/downloads/lotus/storage/sealer/manager_post.go:205"}
2023-03-29T14:48:12.491+0900	ERROR	wdpost	wdpost/wdpost_run.go:414	error generating window post: some sectors (1) were skipped	{"cycle": "2023-03-29T14:36:00.320+0900"}
2023-03-29T14:48:12.499+0900	WARN	wdpost	wdpost/wdpost_run.go:479	generate window post skipped sectors	{"cycle": "2023-03-29T14:36:00.320+0900", "sectors": [{"Miner":1697248,"Number":312269}], "error": "some sectors (1) were skipped", "errorVerbose": "some sectors (1) were skipped:\n    github.com/filecoin-project/lotus/storage/sealer.(*Manager).generateWindowPoSt\n        /home/downloads/lotus/storage/sealer/manager_post.go:205", "try": 1}
2023-03-29T14:48:12.717+0900	INFO	wdpost	wdpost/wdpost_run.go:397	running window post	{"cycle": "2023-03-29T14:36:00.320+0900", "chain-random": "wdH/AUpdKxwON8Xd9YHo++vflKjszu4Qm7OzO9kCTaM=", "deadline": {"CurrentEpoch":2725392,"PeriodStart":2723731,"Index":28,"Open":2725411,"Close":2725471,"Challenge":2725391,"FaultCutoff":2725341,"WPoStPeriodDeadlines":48,"WPoStProvingPeriod":2880,"WPoStChallengeWindow":60,"WPoStChallengeLookback":20,"FaultDeclarationCutoff":70}, "height": "2725392", "skipped": 11}
2023-03-29T14:48:13.008+0900	WARN	sectors	pipeline/fsm.go:799	sector 325994 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: wrong callToWork mapping for call 0-0-00000000-0000-0000-0000-000000000000; expected seal/v0/precommit/1([[{"ID":{"Miner":1697248,"Number":325994},"ProofType":8},"ccJjop/rEPT9trRMBsUpaqRjSRvWNmcIC6Y2IKe5wpQ=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqglwaz47wfphh6swm53fk6w6235a66aokaudfiac3jng4aza2toea"}}]]]), got ()
2023-03-29T14:48:13.034+0900	INFO	sectors	pipeline/states_failed.go:30	SealPreCommit1Failed(325994), waiting 59.965358967s before retrying