16 bit depthmap extra details

[AugmentedRealityCat]

The video example shown on this reddit post is more detailed than what we get even though it was made with the stable-diffusion-webui-depthmap-script extension.
https://www.reddit.com/r/StableDiffusion/comments/11eiqij/partial_3d_model_from_sd_images_still_in_a_very/

Here is the beginning of that video to give you an idea.
https://user-images.githubusercontent.com/114955776/222047336-ac8facd1-9f3a-4c50-af74-b3ad115c2bc6.mp4

According to the poster, the only difference in code is one that allows the system to keep the whole 16 bits of details, while the current implementation reduces it to 8 bit (which is the color depth used by the Stable Diffusion code). That would mean that even though the file we get is encoded as a 16 bit greyscale PNG, that image would not be showing the depth map data in full details.

That difference in code would be the removal of a division by 255, which I found on the following line (346):

stable-diffusion-webui-depthmap-script/scripts/depthmap.py

Line 346 in 189e30a

img = cv2.cvtColor(np.asarray(inputimages[count]), cv2.COLOR_BGR2RGB) / 255.0

And here is the link to the post where I read about that code modification:

yeah but you lose ton of detail , try to edit the depthmap script in python
find the X-value/ 255 and remove the 255 to get the full 16bit depthmap
there is so much detail
https://www.reddit.com/r/StableDiffusion/comments/11eiqij/comment/jaf4s1l/?utm_source=share&utm_medium=web2x&context=3

Does this make sense and is there any other modification required to get even more detailed 16 bit depthmaps, as per the Reddit example?

[graemeniedermayer]

Line 346 is still the coloured image before the depth prediction.

I'm pretty sure the 16-bit image is implemented correctly (you only get 16-bit if you save as a png). They might have been referring to an earlier version of depthmap. Very early versions appear to always cast the 16-bit image to an 8-bit image.

[AugmentedRealityCat]

Line 346 is still the coloured image before the depth prediction.

That might be entirely my error though - it's the only line I found with the / 255 code in it, so I thought that might be it. But that's just me and I know next to nothing about coding.

I know you are familiar with the code since you extended it in quite a few interesting ways already - with that in mind, in your experience, where in the code would you be looking for if you knew there was indeed such a line that imposes a 8-bit limit on the depth map ?

I know I've been using 16 bit depthmaps from greyscale encoded PNGs, that's for sure, but I must admit it is possible that those depth maps were not showing all the details they could, and that some of the data was crushed somewhere during the process.

Do you also have the impression that the depth maps shown in the Reddit video are particularly well detailed ? Could this be explained by them working in a higher resolution ? Or is the resulting mesh more detailed because it's based on custom pointcloud to mesh code instead of the 3d-photo-inpainting code we use over here ?

I share a good portion of your doubts, but I would be ecstatic if a simple adjustment to a line of code was to make the whole thing so much more precise than it already is.

[graemeniedermayer]

It is pretty impressive! It reminds me of using depthmaps+greyscaled images for example #73 . You can get a lot of extra details by mixing depthmaps with the original greyscaled image. Mixing in grey-scaled images does create some artefacts on eyes (pupil bumps) which is why I think this might be part of their workflow.

I'm a little doubtful about their explanation. I feel like there's some secret sauce they aren't sharing.

[AugmentedRealityCat]

I feel like there's some secret sauce they aren't sharing.

I have the exact same feeling. And it bugs me. A lot !

Could this special sauce be the latest monocular research published last week ?
https://github.com/isl-org/ZoeDepth

The redditor was talking about special calibration and metrics, and that to me could be a clue as the full name of that repo is:
ZoeDepth: Combining relative and metric depth

The other aspect they mentioned was related to photogrammetry, but I believe that is only for combining meshes from multiple views and for extending the mesh in occluded areas, but it does not seem to be applied on the examples shown in the video.

[AugmentedRealityCat]

I used the code search function and discovered another instance of division by 255. Here it is - this time it's in the meshing code, so it would make more sense. It does seem to divide the 3 RGB channels by 255, but then again it might be to determine something else than the mesh relief, like storing color information in a vertex color map.

stable-diffusion-webui-depthmap-script/scripts/inpaint/mesh.py

Line 2243 in 189e30a

colors[..., :3] = colors[..., :3]/255.

What do you think ? Should I try to replace that 255 with a 1 ? (I assume it's better not to replace it with a zero !).

More suspects two other instances here but in the mesh tools script - I guess this is less likely to be involved, but I'm way out of my expertise here: it's code !

stable-diffusion-webui-depthmap-script/scripts/inpaint/mesh_tools.py

Line 168 in 189e30a

input_rgb = image[valid_anchor[0]:valid_anchor[1], valid_anchor[2]:valid_anchor[3]] / 255. * context[..., None]

stable-diffusion-webui-depthmap-script/scripts/inpaint/mesh_tools.py

Line 918 in 189e30a

feat[:, node[0] - x_anchor[0], node[1] - y_anchor[0]] = np.array(mesh.nodes[node]['color']) / 255.

[merecesarchviz]

Is possible to have 24Bits of depth? I notice that the script Depth img2img mask get the depth maps with 24bits!

[AugmentedRealityCat]

Is possible to have 24Bits of depth?

Most of the time (99.9%) when I see a reference to 24 bit it refers to 3x8bit for RGB, so it really is 8bpc (8 bit per channel, with 3 channels).

But it's entirely possible in theory to encode this data in 24 bits, it's just not a very common format, particularly for images, which is the type of format I am familiar with.

Here is some context:

For depth maps and other computational passes where the data precision is more important than how they look, like normal maps, or motion vectors, 32 bits per channel (32bpc) is recommended in most cases as it's much more precise. So you could say that is really 96 bits since we have 3 channels with 32 bit in each. 32 bit is also commonly used for HDR images as we need that extra data to express the lighting values that are out of the range covered by 8 bits.

When we refer to 16 bit PNG, those are specially encoded images that only work with greyscale data. It's 16 bit per channel, but it has a single channel, so 1x16bit.

And just to make sure things will remain confusing, there is the case where you have 8 bit per channel (256 values per channel) but instead of having 3 channels, 1 for R, 1 for G and 1 for B, for a total of 24 bits (like your question), you have a fourth channel for transparency (the famous alpha channel) which brings the total to 4x8=32 bits !

That's where it's most important to get precisely what is meant for 32 bit color, while for most other commonly used values (8, 16, 24, 64, 96, 128, 256) refer to a single configuration, 32 bit could mean 32 per channel, or 3x8 bit per channel.

TLDR: it's theoretically possible, but in this case it probably means 8 bit per channel, so the stantdard 0-255 RGB values.

[AugmentedRealityCat]

I'm pretty sure the 16-bit image is implemented correctly (you only get 16-bit if you save as a png). They might have been referring to an earlier version of depthmap. Very early versions appear to always cast the 16-bit image to an 8-bit image.

I think I've found exactly what this was referring to ! I was documenting a bug in the Zoe-Depth extension for Automatic1111-WebUI and as I was finishing it, it occurred to me that the code I was identifying as a problem was a multiplication by 256 !

Here is the bug report
sanmeow/a1111-sd-zoe-depth#2 (comment)

And if you haven't tested it, go install that extension right now, @sanmeow has done a tremendously good job ! The 3d panorama is exactly what I've been doing manually, but with the extra quality provided by ZoeDepth.
https://github.com/sanmeow/a1111-sd-zoe-depth