Missing weights for depth estimation.

[JiuTongBro]

Hi, thanks for your impressive work firstly.

However, when I try to add an instance to the scene, the code threw an error:
FileNotFoundError: [Errno 2] No such file or directory: '.cache/dpt/omnidata_dpt_depth_v2.ckpt'

I suppose it is because the weights for depth estimation is missed. I tried to download it manually and put it under the utils/cache/dpt/ folder, but it still doesn't work. Do you know how to solve it?

This is the full logs about the error, it happened after the Wonder3D reconstruction.

Traceback (most recent call last):
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1739, in write_video
    writer.add_image(image)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1648, in add_image
    if self._proc.stdin.write(data) != len(data):
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train_from_mesh.py", line 261, in <module>
    cams, images, camera_extent = render_multiview_images_from_mesh(args.mesh,
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/threestudio/utils/render.py", line 150, in render_multiview_images_from_mesh
    write_video(save_path, frames, fps=2)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1739, in write_video
    writer.add_image(image)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1610, in __exit__
    self.close()
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1661, in close
    raise RuntimeError(f"Error writing '{self.path}': {s}")
RuntimeError: Error writing '/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/tmp_add/mesh_rendering.mp4': /home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/bin/ffmpeg: error while loading shared libraries: libopenh264.so.5: cannot open shared object file: No such file or directory

/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/timm/models/_factory.py:117: UserWarning: Mapping deprecated model name vit_base_resnet50_384 to current vit_base_r50_s16_384.orig_in21k_ft_in1k.
  model = create_fn(
Traceback (most recent call last):
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 449, in wrapped
    inner()
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 257, in <lambda>
    lambda: self._handle_incoming_message(client_id, message)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 94, in _handle_incoming_message
    cb(client_id, message)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/_gui_api.py", line 157, in _handle_gui_updates
    cb(GuiEvent(client, client_id, handle))
  File "webui.py", line 511, in _
    self.add(self.camera)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "webui.py", line 1371, in add
    depth_estimator = DPT(get_device(), mode="depth")
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/threestudio/utils/dpt.py", line 1023, in __init__
    checkpoint = torch.load(path, map_location="cpu")
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/serialization.py", line 986, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/serialization.py", line 435, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/serialization.py", line 416, in __init__
    super().__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '.cache/dpt/omnidata_dpt_depth_v2.ckpt'

[buaacyw]

Thanks for your interest.
It supposed to be automatically downloaded from online.
I'm not sure why the download failed in your case.

[heheyas]

Hi, JiuTongBro

You can try downloading the checkpoint file directly using the following commands:
If necessary:

mkdir -p .cache/dpt
pip install gdown

Download pre-trained models

gdown '1Jrh-bRnJEjyMCS7f-WsaFlccfPjJPPHI&confirm=t' -O ./cache/dpt

Reference

[JiuTongBro]

Hi, JiuTongBro

You can try downloading the checkpoint file directly using the following commands: If necessary:

mkdir -p .cache/dpt
pip install gdown

Download pre-trained models

gdown '1Jrh-bRnJEjyMCS7f-WsaFlccfPjJPPHI&confirm=t' -O ./cache/dpt

Reference

Thanks, it works, but then the pipeline broke again when reconstructing Gaussian from the mesh. Here is the full log:

Traceback (most recent call last):
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1739, in write_video
    writer.add_image(image)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1648, in add_image
    if self._proc.stdin.write(data) != len(data):
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train_from_mesh.py", line 261, in <module>
    cams, images, camera_extent = render_multiview_images_from_mesh(args.mesh,
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/threestudio/utils/render.py", line 150, in render_multiview_images_from_mesh
    write_video(save_path, frames, fps=2)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1739, in write_video
    writer.add_image(image)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1610, in __exit__
    self.close()
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1661, in close
    raise RuntimeError(f"Error writing '{self.path}': {s}")
RuntimeError: Error writing '/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/tmp_add/mesh_rendering.mp4': /home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/bin/ffmpeg: error while loading shared libraries: libopenh264.so.5: cannot open shared object file: No such file or directory

/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/timm/models/_factory.py:117: UserWarning: Mapping deprecated model name vit_base_resnet50_384 to current vit_base_r50_s16_384.orig_in21k_ft_in1k.
  model = create_fn(
/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torchvision/transforms/functional.py:1603: UserWarning: The default value of the antialias parameter of all the resizing transforms (Resize(), RandomResizedCrop(), etc.) will change from None to True in v0.17, in order to be consistent across the PIL and Tensor backends. To suppress this warning, directly pass antialias=True (recommended, future default), antialias=None (current default, which means False for Tensors and True for PIL), or antialias=False (only works on Tensors - PIL will still use antialiasing). This also applies if you are using the inference transforms from the models weights: update the call to weights.transforms(antialias=True).
  warnings.warn(
depth valid percent:  tensor(0.9923, device='cuda:0')
Traceback (most recent call last):
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 449, in wrapped
    inner()
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 257, in <lambda>
    lambda: self._handle_incoming_message(client_id, message)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 94, in _handle_incoming_message
    cb(client_id, message)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/_gui_api.py", line 157, in _handle_gui_updates
    cb(GuiEvent(client, client_id, handle))
  File "webui.py", line 511, in _
    self.add(self.camera)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "webui.py", line 1436, in add
    new_object_gaussian.load_ply(gs_path)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/gaussiansplatting/scene/vanilla_gaussian_model.py", line 293, in load_ply
    plydata = PlyData.read(path)
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/plyfile.py", line 158, in read
    (must_close, stream) = _open_stream(stream, 'read')
  File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/plyfile.py", line 1345, in _open_stream
    return (True, open(stream, read_or_write[0] + 'b'))
FileNotFoundError: [Errno 2] No such file or directory: '/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/tmp_add/inpaint_gs.obj'

I checked the tmp_add folder, there are four items: multiview_pred_images (a folder), inpaint_mesh.obj, inpainted.png, and removed_bg.png. I suppose it is caused by a failure in train_from_mesh.py, which convert the 3D mesh reconstructed by Wonder3D to Gaussians. Could you please check the code and fix this potential bug? Thanks.

[heheyas]

This error is related to ffmpeg (used inside mediapy), you can try reinstall ffmepg.

Reference

[JiuTongBro]

This error is related to ffmpeg (used inside mediapy), you can try reinstall ffmepg.

Reference

Yes, I have fixed it and successfully run the whole pipeline. Thanks!