Smoldot panics at `inject_key`

[JGiter]

After runtime upgrade we let chain run for some time and at specific blocks, in which some intensive calcualtion happens, Chopsticks panics

[17:27:14.842] INFO (block-builder): Energy Web X building #4,289,093
    app: "chopsticks"
    number: 4289093
    extrinsics: []
    umpCount: 0
        chopsticks::executor  TRACE: [120] Calling BlockBuilder_apply_extrinsic
        chopsticks::executor  TRACE: [120] Completed BlockBuilder_apply_extrinsic
        chopsticks::executor  TRACE: [121] Calling BlockBuilder_apply_extrinsic
        chopsticks::executor  TRACE: [121] Completed BlockBuilder_apply_extrinsic
        chopsticks::executor  TRACE: [122] Calling BlockBuilder_finalize_block
panicked at /home/runner/work/chopsticks/chopsticks/vendor/smoldot/lib/src/executor/runtime_call.rs:839:21:
assertion failed: key.starts_with(req.prefix().as_ref())

Stack:

Error
    at imports.wbg.__wbg_new_abda76e883ba8a5f (@acala-network/chopsticks-executor/dist/cjs/chopsticks_executor.js:749:19)
    at wasm://wasm/00bdc52e:wasm-function[1234]:0x1ba882
    at wasm://wasm/00bdc52e:wasm-function[1421]:0x1c2699
    at wasm://wasm/00bdc52e:wasm-function[1431]:0x1c297d
    at wasm://wasm/00bdc52e:wasm-function[104]:0xb086f
    at wasm://wasm/00bdc52e:wasm-function[497]:0x16e802
    at wasm://wasm/00bdc52e:wasm-function[1491]:0x1c34ad
    at __wbg_adapter_48 (@acala-network/chopsticks-executor/dist/cjs/chopsticks_executor.js:419:10)
    at real (@acala-network/chopsticks-executor/dist/cjs/chopsticks_executor.js:401:22)

What could potentially cause this? Does this point to incorrect migration or more looks like some logical error? I can share code if that helps though it might take some efforts to understand it

[xlc]

are you iterating storages in on_finalize? any runtime code you suspect might be related?

[JGiter]

We only have on_initialize and on_idle hooks. The failed assertion is in ExternalStorageClearPrefix VM state, which suggests that something wrong happens with storage, while it has been cleared. In on_idle we are clearing one StorageDoubleMap with clear_prefix and in on_initialize we are inserting new item in the same map, but it shouldn't be a problem as far as I understand, because first keys in both operations are different

[PanGan21]

Hi @xlc thanks for responding. It would be great to understand if this is some behavior caused from chopsticks or from our code.
As @JGiter mentioned we only use on_initialize and on_idle. Could it be that on_finalize is throwing error because most of the block weight has been consumed already from on_initialize?

[xlc]

on_idle is called during finalization so that could be the cause. can you point your on_idle hook?
and share wasm and reproduce steps so we can debug this