Video textures on Model Viewer

[sercandemirtas]

Hello friends,

Is it possible to add video texture to model viewer?

Im using blender to export glb. I added video texture to my object and added simple animations to keyframes. When I played animation on Blender video texture and animation works well. On model viewer animations works but texture of object just shows first frame. Do I have to use specific video format or Model Viewer does not support video textures ?

[elalish]

No, video textures are not part of the glTF spec, and hence are not supported by us. You could put this feature request to the Khronos 3D formats working group.

[antont]

Would an alternative be to somehow hack a video in the browser to be shown where the model is displayed? I'd need this to show a video in AR. I don't need to use glTF for it otherwise.

Apparently you use three.js so maybe I could find the object in the three scene and put the video there. Or just use some more vanilla way to use WebXR when don't actually need a 3D model viewer..

[elalish]

Yes, three.js supports video textures, so you'll just need to hack down into our private [$scene] symbol member and then do what you like. Of course, you're outside the public API, so there be dragons. No guarantee I won't break you. 😁

[ardboer]

Hi @antont, did you get this to work?

[elalish]

I'm happy to report that video texture support is coming in the next release: #3886

[rmhall]

I have seen requests for video as texture come up several times now, I can count at least 4-5 related discussions. There are also a handful of requests or inquiries about getting access to the Three.js constructors being used, to get direct access to the Three.js scene, groups, etc. @elalish has given some great tips - this thread in particular: #1873 was very useful. Also, the consistency of the notes, warnings and caveats about private symbols and the ramifications on pursuing this or - that things could break at any time is appreciated. I also understand the logic and reasoning noted above over the lack of alignment/support for video as textures in the glTF spec, as well as the fact that it won't directly carry over to Scene Viewer or QuickLook since its not supported there either. All makes complete sense.

That said, I still have some use cases for doing both: video textures and direct access to the inner scene. So I spent some time last week and got both of those working pretty well for my purposes. I am now able to use the VideoTexture feature of Three.js in an interesting way in conjunction with ModelViewer, as well as add primitives and other meshes intro group in the existing scene. It may not be suitable for all use cases, but for my current needs/projects it fits the bill really well. I came up with a very straightforward approach and example along with notes, caveats and other gotchas that come along for the ride, and even how to avoid some (to the extent they can be). It is VERY model specific, so that and the warnings @elalish has provided (Watch out for the dragons!) should be heeded, but I believe it would still be a valuable example to share, even if just to point out the current limitations, and to have a nice tangible reference to share. I would be happy to put a little formality and polish on it and get it published and share back here to aid any other adventurous souls who are interested in pursuing and also to open that up further for any critiques, pointers, improvements - especially if I am missing something even more egregious than the additional caveats I have uncovered.

For now I've attached a quick video showing some of this in action: A ModelViewer loaded .glb (of a custom hardware product with a screen), where I am then gaining access to the Scene, create a new THREE.VideoTexture from an external video (in this case a little clip from Buckaroo Bonzai featuring our national treasure John Lithgow), add it to a material constructor from ModelViewer, create a new THREE.PlaneGeometry, then create a new THREE.Mesh with that new geometry and videoTexture material, set the position and add it to the existing mesh group in the scene so it rotates, zooms, etc. And a couple buttons to trigger play/pause and mute of the accompanying audio. Again a specific use case tied to a specific model, but it works.

vidtexture_mv_poc.mp4

In diving into this, it also helped me understand a bit better on how some things are structured and my biggest take away, revolved around some of the things noted in that same thread on getting direct access to the Three.js scene. This has led me to think about that a bit more, and I know it may not align fully with the projects goals, especially video texture, but setting that specific feature aside for a moment, exposing and getting the direct access to Three.js constructors seems like a logical consideration to make easier, It would be fantastic if it was supported directly, but even if its an unsupported formally or in the release builds/dist(s), but as a nice option for custom builds, given how it opens up more advanced use cases. Imagine setting a directive or environment option(s) or two, or a dedicated "npm run build-with-three-exposed" that aids others in creating a custom build that anyone could run with on their own in an unsupported fashion to bring in additional Three.Js constructors. I was curious if there had been any other thoughts on this topic beyond the mixin/symbol import approach and if anyone had any additional thoughts as to any other viable paths or how this might be different/easier/up for consideration (or not) for version 2.0 of ModelViewer?

[elalish]

Also, see #3886

[Cebolitoss]

@rmhall and @elalish thanks for the solution!
Any chances to have audio in the video? I mean I was able to run the video on my model, but it doesn't play the audio, only the image.
Am I missing something ??

Other thing, does it work on IOs? or only on Android?

[elalish]

See #4068 (reply in thread)

[flommmm]

@rmhall

Hey, maybe I misunderstand the topic here or am offtopic but hopefully you can help me out. Is there any way to have a video texture in a GLB-file which shall be displayed in augmented reality when opening the GLB-file on a smartphone? Which software is able to do that?

[elalish]

There isn't an glTF extension for video textures yet, but one is in the works. In the meanwhile, you can apply them after loading in MV and get everything you desire (at least for Android AR, via WebXR): https://modelviewer.dev/examples/scenegraph/#animatedTexturesExample