KeyError in cache_update

[v3i1y]

Custom Node Testing

• I have tried disabling custom nodes and the issue persists (see how to disable custom nodes if you need help)

Expected Behavior

no errors

Actual Behavior

I started to recently get following errors randomly after upgrading to the latest comfyui

    self._target(*self._args, **self._kwargs)

  File "/root/comfy/ComfyUI/main.py", line 195, in prompt_worker

    e.execute(item[2], prompt_id, item[3], item[4])

  File "/root/comfy/ComfyUI/execution.py", line 655, in execute

    asyncio.run(self.execute_async(prompt, prompt_id, extra_data, execute_outputs))

  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 190, in run

    return runner.run(main)

           ^^^^^^^^^^^^^^^^

  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 118, in run

    return self._loop.run_until_complete(task)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/conda/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete

    return future.result()

           ^^^^^^^^^^^^^^^

  File "/root/comfy/ComfyUI/execution.py", line 701, in execute_async

    result, error, ex = await execute(self.server, dynamic_prompt, self.caches, node_id, extra_data, executed, prompt_id, execution_list, pending_subgraph_results, pending_async_nodes)

                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/root/comfy/ComfyUI/execution.py", line 409, in execute

    execution_list.cache_update(unique_id, caches.outputs.get(unique_id))

  File "/root/comfy/ComfyUI/comfy_execution/graph.py", line 220, in cache_update

    self.execution_cache[to_node_id][node_id] = value

    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^

KeyError: '7'

I saw the code in the error is from some very recent commit, that might have introduced some bug?

Steps to Reproduce

it happens randomly not really sure how to reproduce consistently

Debug Logs

provided above

Other

No response

[telestratium]

I had to do this to get it to work again. my KeyError was 625
#10377 (comment)

[DoctorPavel]

I've also been seeing this type of cache_update related KeyError issue ever since the update before the last one. At first it was KeyError: '7' and since the last update it turned into KeyError: '397'.

[ech3lon42]

Facing the same Issue. It started after updating comfyUI and some python packages (in particular transformers). I also changed manager security_level in config.ini to weak. But I already changed it back to normal and the problem persists.

The first generation always goes through normally. The issue always happens in the second generation, especially when only changing part of the workflow / graph.

Here is a log:

got prompt
Failed to validate prompt for output 267:
* (prompt):
  - Required input is missing: images
* PreviewImage 267:
  - Required input is missing: images
Output will be ignored
FETCH ComfyRegistry Data: 85/101
Using xformers attention in VAE
Using xformers attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
FETCH ComfyRegistry Data: 90/101
Using scaled fp8: fp8 matrix mult: False, scale input: False
Requested to load QwenImageTEModel_
loaded completely 9.5367431640625e+25 7909.737449645996 True
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cuda:0, dtype: torch.float16
FETCH ComfyRegistry Data: 95/101
Requested to load WanVAE
loaded completely 10103.571140289307 242.02829551696777 True
model weight dtype torch.bfloat16, manual cast: None
model_type FLUX
VRAM > 15GiB, disabling CPU offload
Requested to load NunchakuQwenImage
FETCH ComfyRegistry Data: 100/101
FETCH ComfyRegistry Data [DONE]
[ComfyUI-Manager] default cache updated: https://api.comfy.org/nodes
FETCH DATA from: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json [DONE]
[ComfyUI-Manager] All startup tasks have been completed.
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:03<00:00,  2.32it/s]
[ReActor] 14:08:54 - STATUS - Working: source face index [0], target face index [0]
[ReActor] 14:08:54 - STATUS - Using Loaded Source Face Model...
[ReActor] 14:08:54 - STATUS - Analyzing Target Image...
[ReActor] 14:08:58 - STATUS - Swapping...
[ReActor] 14:08:59 - STATUS - --Done!--
[ReActor] 14:08:59 - STATUS - Restoring with GFPGANv1.4.pth | Face Size is set to 512
Requested to load NunchakuQwenImage
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:05<00:00,  1.42it/s]
[ReActor] 14:09:17 - STATUS - Working: source face index [0], target face index [0]
[ReActor] 14:09:17 - STATUS - Using Loaded Source Face Model...
[ReActor] 14:09:17 - STATUS - Analyzing Target Image...
[ReActor] 14:09:18 - STATUS - Swapping...
[ReActor] 14:09:19 - STATUS - --Done!--
[ReActor] 14:09:19 - STATUS - Restoring with GFPGANv1.4.pth | Face Size is set to 512
Requested to load NunchakuQwenImage
100%|████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:05<00:00,  1.43it/s]
Prompt executed in 70.48 seconds
got prompt
Failed to validate prompt for output 267:
* (prompt):
  - Required input is missing: images
* PreviewImage 267:
  - Required input is missing: images
Output will be ignored
Exception in thread Thread-25 (prompt_worker):
Traceback (most recent call last):
  File "threading.py", line 1045, in _bootstrap_inner
  File "threading.py", line 982, in run
  File "I:\StableDiffusion\ComfyNew\ComfyUI_windows_portable\ComfyUI\main.py", line 195, in prompt_worker
    e.execute(item[2], prompt_id, item[3], item[4])
  File "I:\StableDiffusion\ComfyNew\ComfyUI_windows_portable\ComfyUI\execution.py", line 655, in execute
    asyncio.run(self.execute_async(prompt, prompt_id, extra_data, execute_outputs))
  File "asyncio\runners.py", line 190, in run
  File "asyncio\runners.py", line 118, in run
  File "asyncio\base_events.py", line 653, in run_until_complete
  File "I:\StableDiffusion\ComfyNew\ComfyUI_windows_portable\ComfyUI\execution.py", line 701, in execute_async
    result, error, ex = await execute(self.server, dynamic_prompt, self.caches, node_id, extra_data, executed, prompt_id, execution_list, pending_subgraph_results, pending_async_nodes)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "I:\StableDiffusion\ComfyNew\ComfyUI_windows_portable\ComfyUI\execution.py", line 409, in execute
    execution_list.cache_update(unique_id, caches.outputs.get(unique_id))
  File "I:\StableDiffusion\ComfyNew\ComfyUI_windows_portable\ComfyUI\comfy_execution\graph.py", line 220, in cache_update
    self.execution_cache[to_node_id][node_id] = value
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: '326'

[v3i1y]

I think what's the worst is that once this happens the job seems stuck, it's in the pending state forever and there's no way to cancel it. You have to restart comfyui.

[ech3lon42]

"Solution" for me was to use update/update_comfyui_stable.bat to go back to the latest stable branch.

I did try the git update approach and ran "git update-git-for-windows" but even though it updated, this didn't result in any change. Only going back to the stable branch helped in my case.

[loominati69]

"Solution" for me was to use update/update_comfyui_stable.bat to go back to the latest stable branch.

I did try the git update approach and ran "git update-git-for-windows" but even though it updated, this didn't result in any change. Only going back to the stable branch helped in my case.

How to use that command ? Where do i need to run it from

[ech3lon42]

When using ComfyUI portable version, there should be an update folder in your main installation directory and there you will find the mentioned bat file.

As for the "git update-git-for-windows" command, you can run it on windows terminal or git bash. But for me updating git didn't help.

[Kosinkadink]

People experiencing this issue, do you use any startup arguments such as cache-none?

[ech3lon42]

People experiencing this issue, do you use any startup arguments such as cache-none?

.\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --use-sage-attention

is what I use.

[rattus128]

If anyone is able to share a workflow, even it if its an intermittent reproducer that would defiantly help. If there a sense of what node was executing when it crashed, and what nodes are input and output that is invaluable.

[v3i1y]

caption.json

It happened for me in this workflow. Sorry it has a lot custom nodes I wrote myself that I won't be able to share but I think it might have something todo with nodes with lazy inputs

[clavar]

MathExpression|pysssss node crashes the cache system aswell. Using primite int works fine.

[rattus128]

MathExpression|pysssss node crashes the cache system aswell. Using primite int works fine.

Can I get your backtrace? Press show report and paste everything here if possible.