Introducing Marigold 🌼 - a universal monocular depth estimator.

[hdvrai]

Would this be of interest for this plug-in?

https://github.com/prs-eth/Marigold

[graemeniedermayer]

This is a super interesting project! Fine-tuning SD for depth. It also doesn't look like too challenging of an integration.

[semjon00]

Wow, this thing is just fantastic! I can't allocate enough time to implement this, but I would be excited to see this integration implemented! Somebody please help :)

[aulerius]

Wow, and their results are phenomenal for a wide range of scales and scenes.

[hdvrai]

https://github.com/zhyever/PatchFusion is this the same thing?

[aulerius]

This looks more similar to multi-resolution merging that's already implemented in this extension/script.

[bbecausereasonss]

Came here to post this!

[semjon00]

Here is the start... With some probability I will return to this only after some considerable time.
https://github.com/thygate/stable-diffusion-webui-depthmap-script/tree/marigold

[affromero]

Hey! I updated yours here #385

[petermg]

Thanks for this. Got this installed in SD. Seems to work well and as expected.

[petermg]

Hey I was getting inconsistent results using Marigold until I saw this:
from:
https://github.com/prs-eth/Marigold
--seed: Random seed can be set to ensure additional reproducibility. Default: None (using current time as random seed).
Is there any way you can update your branch to include a SEED option in the gui? This way I can try the same seed for video since on video currently with the "random seed" the depth is very jittery. Having the seed option able to be set in the gui I'm hoping would fix this jitter issue.

[semjon00]

@petermg Great idea! I think I'd need to bump the Marigold version first, this would take some time.

[311-code]

I am currently using a blender 2D plane with displacement map and blender's VR mode to view this Marigold depthmap stuff in VR and it's very good.

Using a Quest 3 with Virtual Desktop if you set a green background in your AI generation the objects will appear to be in your room in Mixed Reality if you turn on the VD passthrough mode. There seems to be about a 150 degree view around the person or object. Blender has full motion controls also for moving the it around and rotating. The blender VR plugin seems to work pretty well now for images.

Currently trying to convert this VR unity project which just added Marigold in the beta (though I can't get the Marigold working at the moment due to whl errors) https://github.com/parkchamchi/DepthViewer/releases The source Unity project is available for download there, it also has no positional tracking or motion controls for whatever reason but I'm working on that.

It does realtime 2D video to VR conversion which is pretty good with the basic Midas when I checked it out with a Stable Diffusion Video I made. I'm not sure if the Marigold can do this in realtime though.

My idea here is to add automatic1111 --api access to the Unity Project and be able to type in VR and just have the AI images appear in 3D front of you in mixed reality in the room.

Right now it takes me about 3 minutes to create an image with depth map then update it in Blender for viewing in VR for me currently, a VR Unity build automating all of this would be so much better. Seeing this stuff in full 3D in VR is with the Marigold update pretty mind blowing compared to just looking at your images on a flat screen.

Edit: Also Kijaj added a comfyui workflow so you can update the depthmap in realtime by turning on autoqueue https://github.com/kijai/ComfyUI-Marigold Would love to see something like this on the extension here for auto1111 if dev gets time and it's feasable.

[graemeniedermayer]

One really big thing I've found with marigold is by editing the initial noise you can actually create much lower flickering. I was trying to in-paint but I think I messed up a few aspects so I got this (The outside is suppose to be static):

out1.mp4

This gives a much less flickering depth video which is something that a lot of people seem interested in.

sadtalker.mp4

[petermg]

Very nice result! How do you "editing the initial noise"?

[graemeniedermayer]

I was trying to do inpainting and failed but here's my attempt https://github.com/graemeniedermayer/Marigold-Inpainting .

It's largely in the marigold_pipeline.py file . I was replacing
depth_noise = torch.randn( [1, 4, 96, 96], device=device, dtype=self.dtype )

By masking out section with an actual depthmap.