Report on middle point about unexpected VRAM Problem from yesterday morning for you guys - _ -;;;

[Andypwk]

Expected Behavior

First issues : #7390

So, I'm gonna keep talking about this.

Sorry in advance, English is not my mother language but I believe you guys all understand what I talk to you bros.

ok, let's go.

After installing Windows OS, I installed basic user programs including drivers and then I installed comfyui and tried testing it.

First, basic comfyui test (you know that stupid look glass bottle.)

No problem~ (fortunately) good...?

But!!! This was my big mistake!!!

I shoudn't have trust comfyui node process till I get see all my workflow test result!

Another my full functional WF is not work!

Summary : VRAM still have no-clean problem after one process.

So I recorded this history this below. Just see this below for your fun if you are free.

When I doing nothing with my computer I tested CMD : Nvidia-smi command to see VRAM state

Ah, my vc is 3080ti

ok, looks good as you know. Vram is almost nothing~ less 2% charged.

So I turn on the Comfyui.

Vram is good. No problem~ (yet - _ -;;;)

So I did run que button. ok let's go~

• _ -???

as you see... something goes wrong.

There is a red-line I checked to show seperated cmd message from 'booting endup' and 'new task started'

Vram was 12% when it begun and it always worked without any problem.

but... now it's starting with that weird message

"
K:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\cuda\memory.py:391: FutureWarning: torch.cuda.reset_max_memory_allocated now calls torch.cuda.reset_peak_memory_stats, which resets /all/ peak memory stats. warnings.warn( ..........

"

• _ -???

8months I used and tested this stupid comfyui to learn and know AI image and video generate and

I bet my neck to "I haven's seen that torch memory warning in my life" - _ -;;;;;;;;;;;;;

ok, good. It wasn't first time seeing 'hahaha you sir~ idiot got a new problem' from comfyui alert.

And finally, this (first) try was ended in 84sec.

If this is end-up with just this and no problem, I wouldn't waste my time to post this stupid broken English here.

This is never gone whatever I tried.

Vram clean, purge vram, cache clean etc even I installed RAM cache clean window app but I cannot remove.

Serious?

ye I know sometimes that VRAM state is keeping previous task file so it does look like that but it always back to 12~15% empty mode wheneve I try new task. literally, set default.

but... Now it's always 50% full of memory and make another job slow in very long time.

One sure thing is

new installed comfyui have no problem.

torch module always show memory problem errors.

and I don't know;;;;;;

Summary is...

Please helpuuuuuuuuu me~~~~~~~~~~ ^ ^;;;;;;;;;;;;;;;;;;;;;;;;;;;;

Actual Behavior

Please helpuuuuuuuuu me~~~~~~~~~~ ^ ^;;;;;;;;;;;;;;;;;;;;;;;;;;;;

Steps to Reproduce

Please helpuuuuuuuuu me~~~~~~~~~~ ^ ^;;;;;;;;;;;;;;;;;;;;;;;;;;;;

Debug Logs

got prompt
K:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\cuda\memory.py:391: FutureWarning: torch.cuda.reset_max_memory_allocated now calls torch.cuda.reset_peak_memory_stats, which resets /all/ peak memory stats.
  warnings.warn(
model weight dtype torch.float16, manual cast: None
model_type EPS
Using xformers attention in VAE
Using xformers attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
end_vram - start_vram: 0 - 0 = 0
#62 [CheckpointLoaderSimple]: 20.39s - vram 0b
end_vram - start_vram: 0 - 0 = 0
#100 [Anything Everywhere]: 0.00s - vram 0b
end_vram - start_vram: 0 - 0 = 0
#180 [Power Lora Loader (rgthree)]: 3.60s - vram 0b
end_vram - start_vram: 0 - 0 = 0
#99 [Anything Everywhere]: 0.00s - vram 0b
end_vram - start_vram: 0 - 0 = 0
#98 [Anything Everywhere]: 0.00s - vram 0b
end_vram - start_vram: 0 - 0 = 0
#21 [ttN text]: 0.00s - vram 0b
end_vram - start_vram: 0 - 0 = 0
#20 [ttN text]: 0.00s - vram 0b
end_vram - start_vram: 0 - 0 = 0
#18 [ttN text]: 0.00s - vram 0b
=============================================================
Portrait Master positive prompt:
portrait of a girl. looking at viewer. background is white. close up eye <lora:fix\background\white_1_0.safetensors:1.0>, a plain white background, highly detailed, (woman 18-years-old:1.5), (almond eyes shape:1.05), (wavy cut hairstyle:1.05), (long:1.05), (professional photo, balanced photo, balanced exposure:1.2)

Portrait Master negative prompt:
(watermark:1.2), (text:1.2), (logo:1.2), (3d render:1.2), drawing, painting, crayon, cartoon, painting, illustration, (worst quality, low quality, normal quality:2) busy background, dark background, patterned background, (shinny skin, shiny skin, reflections on the skin, skin reflections:1.35)
=============================================================
end_vram - start_vram: 0 - 0 = 0
#16 [PortraitMaster]: 0.00s - vram 0b
end_vram - start_vram: 0 - 0 = 0
#17 [easy showAnything]: 0.00s - vram 0b
end_vram - start_vram: 0 - 0 = 0
#178 [SeargeIntegerPair]: 0.00s - vram 0b
end_vram - start_vram: 0 - 0 = 0
#12 [EmptyLatentImage]: 0.00s - vram 0b
end_vram - start_vram: 0 - 0 = 0
#90 [LoadImage]: 0.05s - vram 0b
end_vram - start_vram: 0 - 0 = 0
#192:0 [Load Image Batch]: 0.00s - vram 0b
end_vram - start_vram: 0 - 0 = 0
#192:1 [easy imageSwitch]: 0.00s - vram 0b
Requested to load SDXLClipModel
loaded completely 9652.8 1560.802734375 True
end_vram - start_vram: 1679529340 - 0 = 1679529340
#10 [CLIPTextEncode]: 4.45s - vram 1679529340b
end_vram - start_vram: 1679529340 - 1645139976 = 34389364
#9 [CLIPTextEncode]: 0.12s - vram 34389364b
end_vram - start_vram: 1645139976 - 1645139976 = 0
#194:0 [LoadImage]: 0.01s - vram 0b
end_vram - start_vram: 1645139976 - 1645139976 = 0
#194:1 [Load Image Batch]: 0.00s - vram 0b
end_vram - start_vram: 1645139976 - 1645139976 = 0
#194:2 [easy imageSwitch]: 0.00s - vram 0b
end_vram - start_vram: 1645139976 - 1645139976 = 0
#193:0 [LoadImage]: 0.02s - vram 0b
end_vram - start_vram: 1645139976 - 1645139976 = 0
#193:1 [Load Image Batch]: 0.00s - vram 0b
end_vram - start_vram: 1645139976 - 1645139976 = 0
#193:2 [easy imageSwitch]: 0.00s - vram 0b
end_vram - start_vram: 1645139976 - 1645139976 = 0
#7 [ImageBatch]: 0.00s - vram 0b
end_vram - start_vram: 1645139976 - 1645139976 = 0
#94 [easy imageSwitch]: 0.00s - vram 0b
end_vram - start_vram: 1645139976 - 1645139976 = 0
#4 [ControlNetLoader]: 6.97s - vram 0b
Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with options: {'CUDAExecutionProvider': {'device_id': '0', 'has_user_compute_stream': '0', 'cudnn_conv1d_pad_to_nc1d': '0', 'user_compute_stream': '0', 'gpu_external_alloc': '0', 'gpu_mem_limit': '18446744073709551615', 'enable_cuda_graph': '0', 'gpu_external_free': '0', 'gpu_external_empty_cache': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'cudnn_conv_algo_search': 'EXHAUSTIVE', 'do_copy_in_default_stream': '1', 'cudnn_conv_use_max_workspace': '1', 'tunable_op_enable': '0', 'tunable_op_tuning_enable': '0', 'tunable_op_max_tuning_duration_ms': '0', 'enable_skip_layer_norm_strict_mode': '0', 'prefer_nhwc': '0', 'use_ep_level_unified_stream': '0', 'use_tf32': '1', 'sdpa_kernel': '0', 'fuse_conv_bias': '0'}, 'CPUExecutionProvider': {}}
find model: K:\ComfyUI_windows_portable\ComfyUI\models\insightface\models\antelopev2\1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with options: {'CUDAExecutionProvider': {'device_id': '0', 'has_user_compute_stream': '0', 'cudnn_conv1d_pad_to_nc1d': '0', 'user_compute_stream': '0', 'gpu_external_alloc': '0', 'gpu_mem_limit': '18446744073709551615', 'enable_cuda_graph': '0', 'gpu_external_free': '0', 'gpu_external_empty_cache': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'cudnn_conv_algo_search': 'EXHAUSTIVE', 'do_copy_in_default_stream': '1', 'cudnn_conv_use_max_workspace': '1', 'tunable_op_enable': '0', 'tunable_op_tuning_enable': '0', 'tunable_op_max_tuning_duration_ms': '0', 'enable_skip_layer_norm_strict_mode': '0', 'prefer_nhwc': '0', 'use_ep_level_unified_stream': '0', 'use_tf32': '1', 'sdpa_kernel': '0', 'fuse_conv_bias': '0'}, 'CPUExecutionProvider': {}}
find model: K:\ComfyUI_windows_portable\ComfyUI\models\insightface\models\antelopev2\2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with options: {'CUDAExecutionProvider': {'device_id': '0', 'has_user_compute_stream': '0', 'cudnn_conv1d_pad_to_nc1d': '0', 'user_compute_stream': '0', 'gpu_external_alloc': '0', 'gpu_mem_limit': '18446744073709551615', 'enable_cuda_graph': '0', 'gpu_external_free': '0', 'gpu_external_empty_cache': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'cudnn_conv_algo_search': 'EXHAUSTIVE', 'do_copy_in_default_stream': '1', 'cudnn_conv_use_max_workspace': '1', 'tunable_op_enable': '0', 'tunable_op_tuning_enable': '0', 'tunable_op_max_tuning_duration_ms': '0', 'enable_skip_layer_norm_strict_mode': '0', 'prefer_nhwc': '0', 'use_ep_level_unified_stream': '0', 'use_tf32': '1', 'sdpa_kernel': '0', 'fuse_conv_bias': '0'}, 'CPUExecutionProvider': {}}
find model: K:\ComfyUI_windows_portable\ComfyUI\models\insightface\models\antelopev2\genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0
Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with options: {'CUDAExecutionProvider': {'device_id': '0', 'has_user_compute_stream': '0', 'cudnn_conv1d_pad_to_nc1d': '0', 'user_compute_stream': '0', 'gpu_external_alloc': '0', 'gpu_mem_limit': '18446744073709551615', 'enable_cuda_graph': '0', 'gpu_external_free': '0', 'gpu_external_empty_cache': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'cudnn_conv_algo_search': 'EXHAUSTIVE', 'do_copy_in_default_stream': '1', 'cudnn_conv_use_max_workspace': '1', 'tunable_op_enable': '0', 'tunable_op_tuning_enable': '0', 'tunable_op_max_tuning_duration_ms': '0', 'enable_skip_layer_norm_strict_mode': '0', 'prefer_nhwc': '0', 'use_ep_level_unified_stream': '0', 'use_tf32': '1', 'sdpa_kernel': '0', 'fuse_conv_bias': '0'}, 'CPUExecutionProvider': {}}
find model: K:\ComfyUI_windows_portable\ComfyUI\models\insightface\models\antelopev2\glintr100.onnx recognition ['None', 3, 112, 112] 127.5 127.5
Applied providers: ['CUDAExecutionProvider', 'CPUExecutionProvider'], with options: {'CUDAExecutionProvider': {'device_id': '0', 'has_user_compute_stream': '0', 'cudnn_conv1d_pad_to_nc1d': '0', 'user_compute_stream': '0', 'gpu_external_alloc': '0', 'gpu_mem_limit': '18446744073709551615', 'enable_cuda_graph': '0', 'gpu_external_free': '0', 'gpu_external_empty_cache': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'cudnn_conv_algo_search': 'EXHAUSTIVE', 'do_copy_in_default_stream': '1', 'cudnn_conv_use_max_workspace': '1', 'tunable_op_enable': '0', 'tunable_op_tuning_enable': '0', 'tunable_op_max_tuning_duration_ms': '0', 'enable_skip_layer_norm_strict_mode': '0', 'prefer_nhwc': '0', 'use_ep_level_unified_stream': '0', 'use_tf32': '1', 'sdpa_kernel': '0', 'fuse_conv_bias': '0'}, 'CPUExecutionProvider': {}}
find model: K:\ComfyUI_windows_portable\ComfyUI\models\insightface\models\antelopev2\scrfd_10g_bnkps.onnx detection [1, 3, '?', '?'] 127.5 128.0
set det-size: (640, 640)
end_vram - start_vram: 1645139976 - 1645139976 = 0
#5 [InstantIDFaceAnalysis]: 5.26s - vram 0b
end_vram - start_vram: 1645139976 - 1645139976 = 0
#3 [InstantIDModelLoader]: 5.54s - vram 0b
K:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\insightface\utils\transform.py:68: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
  P = np.linalg.lstsq(X_homo, Y)[0].T # Affine matrix. 3 x 4
INFO: InsightFace detection resolution lowered to (448, 448).
INFO: InsightFace detection resolution lowered to (448, 448).
end_vram - start_vram: 2491896840 - 1645139976 = 846756864
#6 [ApplyInstantID]: 1.52s - vram 846756864b
end_vram - start_vram: 2490802696 - 2490802696 = 0
#199 [JWFloat]: 0.00s - vram 0b
end_vram - start_vram: 2490802696 - 2490802696 = 0
#198 [Sampler Scheduler Settings (JPS)]: 0.00s - vram 0b
end_vram - start_vram: 2490802696 - 2490802696 = 0
#203 [easy float]: 0.00s - vram 0b
end_vram - start_vram: 2490802696 - 2490802696 = 0
#202 [easy int]: 0.00s - vram 0b
Requested to load SDXL
Requested to load ControlNet
loaded completely 7626.188103485108 4897.0483474731445 True
loaded completely 2729.091843414307 2386.120147705078 True
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.70it/s]
end_vram - start_vram: 9053826552 - 2490802696 = 6563023856
#11 [KSampler]: 23.94s - vram 6563023856b
end_vram - start_vram: 8507551936 - 8507551936 = 0
#103 [LatentUpscale]: 0.00s - vram 0b
Requested to load SDXL
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:10<00:00,  1.82it/s]
end_vram - start_vram: 9070143992 - 8507551936 = 562592056
#102 [KSampler]: 11.05s - vram 562592056b
Requested to load AutoencoderKL
loaded completely 299.7635269165039 159.55708122253418 True
end_vram - start_vram: 8523915456 - 8523915456 = 0
#13 [VAEDecode]: 1.33s - vram 0b
end_vram - start_vram: 3494359654 - 3494359654 = 0
#119 [LoadImage]: 0.01s - vram 0b
end_vram - start_vram: 3494359654 - 3494359654 = 0
#158:0 [ColorMatch]: 0.25s - vram 0b
# 😺dzNodes: LayerStyle -> Brightness Contrast V2 Processed 1 image(s).
end_vram - start_vram: 3494359654 - 3494359654 = 0
#158:1 [LayerColor: BrightnessContrastV2]: 0.04s - vram 0b
end_vram - start_vram: 3561321010 - 3494359654 = 66961356
#158:2 [ImageSharpen]: 0.02s - vram 66961356b
end_vram - start_vram: 3494359654 - 3494359654 = 0
#158:3 [ColorAdjust(FaceParsing)]: 0.01s - vram 0b
end_vram - start_vram: 3494359654 - 3494359654 = 0
#14 [PreviewImage]: 0.11s - vram 0b
end_vram - start_vram: 3494359654 - 3494359654 = 0
#82 [SaveImage]: 0.13s - vram 0b
Prompt executed in 84.92 seconds
[LogConsole] client [211a37f61f034e1aa507d14511128d35], console [a651a6c5-46d6-4ac6-9f27-abdacc82c3bb], connected
[LogConsole] client [211a37f61f034e1aa507d14511128d35], console [fa5e287e-9b7c-4c78-95f9-de88357328c9], disconnected

Other

Please helpuuuuuuuuu me~~~~~~~~~~ ^ ^;;;;;;;;;;;;;;;;;;;;;;;;;;;;

[Andypwk]

one more added.

I tried second test.

Now, the memory state is 70%, never go to 20% under.

"
end_vram - start_vram: 1037853734 - 1037853734 = 0
#158 [ColorMatch]: 0.46s - vram 0b

😺dzNodes: LayerStyle -> Brightness Contrast V2 Processed 1 image(s).
end_vram - start_vram: 1037853734 - 1037853734 = 0
#158 [LayerColor: BrightnessContrastV2]: 0.08s - vram 0b
end_vram - start_vram: 1104815090 - 1037853734 = 66961356
#158 [ImageSharpen]: 0.86s - vram 66961356b
end_vram - start_vram: 1037853734 - 1037853734 = 0
#158 [ColorAdjust(FaceParsing)]: 0.02s - vram 0b
end_vram - start_vram: 1037853734 - 1037853734 = 0
#14 [PreviewImage]: 0.18s - vram 0b
end_vram - start_vram: 1037853734 - 1037853734 = 0
#82 [SaveImage]: 0.18s - vram 0b
Prompt executed in 246.13 seconds
"

5times slower too.

[Andypwk]

Third test.

got prompt
K:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\cuda\memory.py:391: FutureWarning: torch.cuda.reset_max_memory_allocated now calls torch.cuda.reset_peak_memory_stats, which resets /all/ peak memory stats.
warnings.warn(
end_vram - start_vram: 8519768 - 8519768 = 0
#318 [LoadImage]: 0.05s - vram 0b
K:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\cuda\memory.py:391: FutureWarning: torch.cuda.reset_max_memory_allocated now calls torch.cuda.reset_peak_memory_stats, which resets /all/ peak memory stats.
warnings.warn(
end_vram - start_vram: 972303600 - 8519768 = 963783832
#294 [easy imageRemBg]: 2.02s - vram 963783832b
end_vram - start_vram: 184823024 - 184823024 = 0
#349 [EmptyImage]: 0.00s - vram 0b
end_vram - start_vram: 184823024 - 184823024 = 0
#262 [easy imageSize]: 0.00s - vram 0b

😺dzNodes: LayerStyle -> ImageScaleByAspectRatio V2 Processed 1 image(s).

end_vram - start_vram: 184823024 - 184823024 = 0
#280 [LayerUtility: ImageScaleByAspectRatio V2]: 0.16s - vram 0b
end_vram - start_vram: 184823024 - 184823024 = 0
#289 [easy imageSize]: 0.00s - vram 0b
end_vram - start_vram: 184823024 - 184823024 = 0
#230 [ImageConcanate]: 0.05s - vram 0b
end_vram - start_vram: 184823024 - 184823024 = 0
#232 [PreviewImage]: 0.11s - vram 0b
end_vram - start_vram: 184823024 - 184823024 = 0
#300 [EmptyImage]: 0.00s - vram 0b
end_vram - start_vram: 184823024 - 184823024 = 0
#350 [EmptyImage]: 0.00s - vram 0b
end_vram - start_vram: 184823024 - 184823024 = 0
#241 [ImageConcanate]: 0.05s - vram 0b
end_vram - start_vram: 184823024 - 184823024 = 0
#242 [PreviewImage]: 0.06s - vram 0b
Using xformers attention in VAE
Using xformers attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
end_vram - start_vram: 184823024 - 184823024 = 0
#203 [VAELoader]: 1.02s - vram 0b
end_vram - start_vram: 184823024 - 184823024 = 0
#243 [ImageToMask]: 0.00s - vram 0b
K:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-GGUF\loader.py:91: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\pytorch\torch\csrc\utils\tensor_numpy.cpp:209.)
torch_tensor = torch.from_numpy(tensor.data) # mmap
gguf qtypes: Q8_0 (169), F32 (50)
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
clip missing: ['text_projection.weight']
end_vram - start_vram: 184823024 - 184823024 = 0
#202 [DualCLIPLoaderGGUF]: 4.57s - vram 0b
Requested to load FluxClipModel_
loaded completely 9378.539028930663 5313.69970703125 True
end_vram - start_vram: 6283028216 - 184823024 = 6098205192
#172 [CLIPTextEncode]: 17.69s - vram 6098205192b
end_vram - start_vram: 5493448440 - 5493448440 = 0
#269 [ConditioningZeroOut]: 0.00s - vram 0b
end_vram - start_vram: 6283028216 - 5493448440 = 789579776
#168 [CLIPTextEncode]: 0.98s - vram 789579776b
end_vram - start_vram: 5493448440 - 5493448440 = 0
#133 [FluxGuidance]: 0.00s - vram 0b
Requested to load AutoencodingEngine
loaded completely 5771.611099243164 159.87335777282715 True
end_vram - start_vram: 5493448440 - 5493448440 = 0
#77 [InpaintModelConditioning]: 9.81s - vram 0b
gguf qtypes: F32 (466), Q5_0 (304), F16 (10)
model weight dtype torch.bfloat16, manual cast: None
model_type FLUX
end_vram - start_vram: 352462390 - 352462390 = 0
#217 [UnetLoaderGGUF]: 0.62s - vram 0b
end_vram - start_vram: 352462390 - 352462390 = 0
#351 [LoraLoaderModelOnly]: 0.28s - vram 0b
end_vram - start_vram: 352462390 - 352462390 = 0
#272 [DifferentialDiffusion]: 0.00s - vram 0b
Requested to load Flux
loaded completely 9046.618929748534 8020.982666015625 True
0%| | 0/25 [00:00<?, ?it/s]

Now it's not work, frized.

Vram is not empty ever.

It was started from 2days ago comfyui update.

I guess there is some problem with torch module.

Because when VRAM is almost full empty 12% like it does, Torch always show new error message (I haven't seen) suddenly from 2days ago updated.

[ltdrdata]

1. A FutureWarning is a message for developers, indicating that the library structure may change in the future. It's not something end users need to worry about, so you can safely ignore it.
2. If you want to ensure that VRAM is always unloaded after a workflow finishes, add the --disable-smart-memory option to the CLI.
3. VRAM usage caused by custom nodes is outside the scope of what ComfyUI can manage.