Is 3D generation within the scope of this project?

[rujialiu]

Altough the techniques are very similar, it still "feels different" from image/video generation, so I'm asking. Open-weight 3D generation models are not as mature as image/video generation, but they're still very useful, and ComfyUI supports 3D generation.

So, is 3D generation within the scope of this project? If the answer is "yes" but not planned due to lack of time etc, at least there is possibility for future community contribution.

[leejet]

Support for the Wan video model has been added. You can check it out here #778.

[rujialiu]

That's awesome news! Thanks a lot! Would you please think about my question? Currently texture generation in open weight models is not mature and quite complex (only Hunyuan3D 2.1+ supports PBR materials but the license is restricted), but we could start with mesh generation, for which we have at least 2 good models: TripoSG and Step1X-3D.

Having read their inference code, I think supporting mesh generation wouldn't bloat stable-diffusion.cpp's codebase because we can only implement core functionalities and let postprocessing done elsewhere. @leejet

[Seas0]

Altough the techniques are very similar, it still "feels different" from image/video generation, so I'm asking. Open-weight 3D generation models are not as mature as image/video generation, but they're still very useful, and ComfyUI supports 3D generation.

So, is 3D generation within the scope of this project? If the answer is "yes" but not planned due to lack of time etc, at least there is possibility for future community contribution.

I think 3D assets generation may require sparse tensor arithmetic, which I doubt ggml would even gain support ever...

[rujialiu]

I think 3D assets generation may require sparse tensor arithmetic, which I doubt ggml would even gain support ever...

Thanks for the reminder. I can implement it 😄

[leejet]

Regarding 3D assets generation, I don't know much about it. I'll look into it later when I have time.

[rujialiu]

Nice to hear that!

BTW: I'm been programming 3D in C++ for 10+ years and I'll be happy to help with non-ML stuffs like preprocessing/postprocessing/rendering etc