`fromPixels` returns invalid values for `webgpu` backend

[vladmandic]

i've noticed some errors during model execution, so went back to check inputs and it seems that fromPixels returns incorrect values for webgpu backend

await tf.setBackend('webgpu');
// input is a canvas which is a copy of a webcam input in 1280x720 resolution
const buffer = await tf.browser.fromPixelsAsync(canvas);
const data = await buffer.data();
for (let idx = 0; idx < data.length; idx++) {
  if (data[idx] < 0 || data[idx] > 255) console.log({ idx, val: data[idx], length: data.length });
}

{idx: 1094922, val: -1, length: 2764800}
{idx: 1094925, val: -1, length: 2764800}
{idx: 2275694, val: -1, length: 2764800}

i cannot think of any scenario where pixel value of -1 is a valid thing to have,
especially on only 3 random pixels out of entire (otherwise valid) image
note that invalid values do not occur on every frame, but it does happen often enough

processing the same input shows no such problems when using webgl backend

environment: tfjs 3.20.0, tfjs-backend-webgpu 0.0.1-alpha.13, chrome canary 107

[qjia7]

Assign to @xhcao, please take a look, thanks!

[xhcao]

@vladmandic , are there any steps that I could reproduce this issue locally? And comparing webgl backend with the same input, how many pixels that are different? Most or a few？

[xhcao]

@vladmandic Or you could give me the input data, record the canvas content(assume it is 2d context canvas) by getImageData() when there is -1 value after fromPixels.

[vladmandic]

digging in a bit into tf.browser.fromPixelsAsync...

• confirmed that it only happens for webgpu and not webgl
• issue only occurs when input is HTMLVideoElement
  • it DOES NOT occur for HMTLCanvasElement
  • for example, copying same video frame to canvas and then using fromPixelsAsync on canvas shows no error
• issue is proportional to video complexity (combination of scene change rate, video resolution and frame rate)
  • low complexity videos show no problems
  • medium ones may have 1-2 error every few hundred frames
  • complex ones have hundreds of errors on each frame

basically, what happens is that after some offset within a frame, pixel values start drifting by -1 in general

so if actual pixel value happens to be 0, it will result in -1 being returned which is an invalid value.

all this leads me to believe this is a webgpu sync issue, but this pixel value drift after some offset is weird...

anyhow, full reproduction: http://vladmandic.github.io/nudenet/src/verify.html
and code is at https://github.com/vladmandic/nudenet/blob/main/src/verify.ts

[qjia7]

The above information reminds me that it may be a bug in importExternalTexture, which is a zero copy path for video. @shaoboyan, who implemented this feature in chrome, any thoughts on this?

@vladmandic I suppose tf.browser.fromPixels has the similar issue. And suggest you switch to fromPixels since we plan to remove fromPixelsAsync in next big version upgrade. The fromPixelsAsync is just a simple wrapper of fromPixels to resolve a webgpu's old API problem, which doesn't exist now.

[shaoboyan]

@vladmandic @qjia7 Thanks for the info. It seems like an issue about importExternalTexture but needs further investigation.

To @qjia7 : As you may know, GPUExternalTexture has a big issue in lifecycle and is being reshaped in WebGPU spec, I think we should disable the import path again (Sry) until the spec is ready again. And we could double check whether this issue is still exists.

To @vladmandic : Thanks for your cases, that's super helpful. Would you mind to point me to a complex one video(e.g. online url) so I could debug natively for this API.

[vladmandic]

@shaoboyan live link i've posted (https://vladmandic.github.io/nudenet/src/verify.html, sorry there was a typo in the url) includes a sample video.
its only 10sec long, but on my system it generates thousands of errors per frame - and it logs to browser console.

example:

{tf: '3.20.0', backend: 'webgpu', available: {…}, flags: {…}}
{mediaTime: 0.64, errors: 44492, videoData: Int32Array(8294400), canvasData: Int32Array(8294400)}

[shaoboyan]

Thanks, I could reproduce this and I think this is due to the GPUExternalTexture.

[qjia7]

@shaoboyan Please file a chromium/dawn bug to trace the GPUExternalTexture issue. And this issue will be blocked by that one.
@vladmandic To continue to use fromPixels with video, you can set flag WEBGPU_IMPORT_EXTERNAL_TEXTURE to false. In this path, it will package the video to canvas and then call copyExternalImageToTexture() to do the following thing.

[shaoboyan]

Chromium issue filed: https://bugs.chromium.org/p/chromium/issues/detail?id=1368945. I'll handle this.

[xhcao]

As @qjia7 comment, Using tf.env().set('WEBGPU_IMPORT_EXTERNAL_TEXTURE', false) could work around this issue, but I did not verify whether there is a performance loss. Let's wait @shaoboyan to fix this issue, if the issue cannot be fixed in a short time, we could set WEBGPU_IMPORT_EXTERNAL_TEXTURE to false by default.