[Intel XPU] Qwen3-VL DeepStack multi-image boolean indexing selects wrong token count (1059 vs 2061); CPU-index workaround #16991

Activity

1. Goldlionren commented on Oct 13, 2026

   

   Goldlionren

   on Oct 13, 2026 – with ChatGPT Codex Connector

   Author

   More actions

   Follow-up: visual-output validation not yet passed

   Important correction to the original report: after applying the XPU integer-index workaround, the two-reference workflow no longer raises the 1059 vs 2061 exception and can generate a PNG; however, the reporter has now inspected the generated image and it is visually corrupted (large checkerboard-like region over the upper ~2/3 of a 1440×896 output, with only the lower region containing coherent image content).

   The PNG's alpha channel is nearly opaque (observed alpha range 250–255), so the checkerboard is embedded RGB content, not viewer transparency. This does not yet prove the workaround caused the corruption. Potential variables still to isolate: input reference images, output/compositing nodes, FP8 text-encoder mode, CK Triton/quantized model paths, and other XPU indexing errors. No clean two-image quality pass should be claimed yet.

   Revised status: structural DeepStack shape mismatch bypassed; multi-image output correctness UNVERIFIED/FAIL for current example. We will compare direct VAE-decoded SaveImage output, single-vs-dual image runs, and reference-image contents before identifying the cause. The upstream PyTorch bug and standalone reproducer remain independently confirmed.
2. Goldlionren commented on Oct 13, 2026

   

   Goldlionren

   on Oct 13, 2026 – with ChatGPT Codex Connector

   Author

   More actions

   Further A/B isolation on the reporter's Arc 140V: restoring stock comfy/text_encoders/llama.py immediately restores correct single-image generation. The earlier XPU branch using index_select(1, indices) plus index_copy_(1, indices, updated) caused severe checkerboard-like image corruption in both single- and two-image workflows, even though a small standalone indexing-mapping test produced exact CPU equality. Toggling CK Triton or FP8 text-encoder mode does not affect the correct single-image output after restoring stock code. Please do not adopt the earlier index_copy_ workaround as a verified image-quality fix. We are testing a more conservative approach: keep the original boolean-index assignment for the working single contiguous image range; for multiple separated image ranges, derive contiguous spans on CPU and write each range with ordinary slice assignment, avoiding both XPU boolean nonzero and index_copy_. End-to-end image quality is not yet verified for this new approach.
3. Goldlionren commented on Oct 13, 2026

   

   Goldlionren

   on Oct 13, 2026 – with ChatGPT Codex Connector

   Author

   More actions

   Update: V2 contiguous-slice workaround passes single-image regression and full dual-image inference (visual quality still to be confirmed)

   The earlier XPU index_select + index_copy_ workaround was rejected after corrupted single-image outputs. The reporter restored stock llama.py: single-image output was correct, regardless of FP8 text-encoder toggle or CK Triton toggle.

   A second workaround now preserves the exact stock boolean-assignment for a single contiguous image span and, only when more than one image span is present, constructs image spans from a CPU copy of the boolean mask and uses basic PyTorch slice assignment rather than XPU boolean advanced indexing or index_copy_.

   Relevant multi-image section of the working local experimental patch (JR-XPU-DEEPSTACK-SLICE-V2):

   # In the DeepStack injection block; assume x.shape == [1, seq, hidden],
   # and validated visual_pos_masks.shape == [1, seq].
   mask_cpu = visual_pos_masks[0].detach().to(device="cpu", dtype=torch.int8)
   padded = torch.cat((mask_cpu.new_zeros(1), mask_cpu, mask_cpu.new_zeros(1)))
   edges = torch.diff(padded)
   starts = (edges == 1).nonzero(as_tuple=True)[0].tolist()
   ends = (edges == -1).nonzero(as_tuple=True)[0].tolist()
   
   if len(starts) <= 1:
       x[visual_pos_masks] = x[visual_pos_masks] + deepstack_embeds[i].to(x)
   else:
       # Validate x/mask shapes, len(starts)==len(ends), 
       # and features.shape == [sum(end-start), x.shape[-1]].
       features = deepstack_embeds[i].to(x)
       cursor = 0
       for start, end in zip(starts, ends):
           count = end - start
           x[:, start:end, :] = (
               x[:, start:end, :]
               + features[cursor:cursor + count].unsqueeze(0)
           )
           cursor += count

   The reporter ran a standalone slice correctness check without errors, then ran real Qwen-Image 2.1 FP8 text-encoder + INT8 ConvRot diffusion workflows:

   • Single image (stock path): seq=1069, image tokens 1025; generated visually correct output; 25/25 steps; 298.56 sec total.
   • Two images (V2 path): seq=2135, image spans [22,1047), [1053,2089), 2061 tokens. The V2 diagnostic printed for both encode passes. Completed 25/25 sampling steps and image generation, 368.98 sec total; no shape exception or crash.
   • Two images at a higher resolution with CK Triton enabled: seq=2136, same image spans; completed 25/25, 500.94 sec total. These times must NOT be used to infer Triton impact because output resolution changed.
   • The reporter has not yet provided side-by-side visual output validation for V2 dual-image outputs. Please treat V2 as an experimental workaround pending two-image image-quality verification, not as a production-ready fix.

   Standalone PyTorch boolean indexing bug: pytorch/pytorch#200713.
4. Goldlionren commented on Oct 13, 2026

   

   Goldlionren

   on Oct 13, 2026 – with ChatGPT Codex Connector

   Author

   More actions

   Final visual validation update — V2 Slice workaround PASSED on Arc 140V

   The reporter has now inspected the actual generated PNG outputs and confirms both the single-image and dual-image outputs are visually correct, with none of the earlier V1 checkerboard/vertical-layout artifacts. This validates the V2 CPU-span + XPU basic slice assignment workaround on the reported Ubuntu 26.04 / Arc 140V / torch 2.14.1+xpu environment (but not other hardware yet).

   Evidence already provided in the previous comment: single image 25/25 steps and 298.56 s; dual image 25/25 steps and 368.98 s, with spans [22,1047), [1053,2089) and 2061 DeepStack tokens. Another higher-resolution dual-image 25/25 pass with CK Triton enabled took 500.94 s; this is not a controlled Triton performance comparison.

   V1 index_select/index_copy_ must not be used due to severe image quality regression. V2 preserves original single-span update, using basic per-span XPU slices only for two or more disjoint image spans. A single-file patch and reproducible rollback/upgrade tooling have been prepared. A PR is pending repository-contents write authorization for the reporter's fork (the current GitHub integration returns HTTP 403 on branch creation); no PR is claimed until a push succeeds.

   Root-cause tracker: pytorch/pytorch#200713.
5. comfyui-wiki

   linked a pull request that will close this issueFix Qwen3-VL DeepStack multi-image writes on Intel XPU #16992on Oct 13, 2026