Skip to content

Commit

Permalink
add sam2_depthanything example (#7306)
Browse files Browse the repository at this point in the history
  • Loading branch information
pablovela5620 authored and nikolausWest committed Aug 29, 2024
1 parent 9bebd1a commit 3731b26
Show file tree
Hide file tree
Showing 2 changed files with 25 additions and 0 deletions.
1 change: 1 addition & 0 deletions examples/manifest.toml
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ examples = [
# display order, most interesting first
"detect_and_track_objects",
"face_tracking",
"sam2_depthanything",
"human_pose_tracking",
"gesture_detection",
"ocr",
Expand Down
24 changes: 24 additions & 0 deletions examples/python/sam2_depthanything/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
<!--[metadata]
title = "Sam2 + DepthAnything2"
tags = ["2D", "3D", "HuggingFace", "Depth", "Pinhole camera", "SAM", "Segmentation"]
source = "https://github.com/pablovela5620/sam2-depthanything"
thumbnail = "https://static.rerun.io/sam2_depthanything/ecc229c54a04c55bfba236f86e15cd285429e412/480w.png"
thumbnail_dimensions = [480, 268]
-->


https://vimeo.com/1003789426?loop=1&autopause=0&background=1&muted=1&ratio=2802:1790

## Background
Segment Anything 2 is follow up work on Segment Anything, that extends the state of the art segmentation capabilities into videos. This is done by adding a per session memory module that captures information about the target object in the video. This allows SAM 2 to track the selected object throughout all video frames, even if the object temporarily disappears from view, as the model has context of the object from previous frames. Depth Anything 2 is a monocular depth estimation model trained on a large amount of synthetic data and real data to achieve state of the art depth estimation. The two models are combined to allow tracking an object in 3D from just a single monocular video!

## Run the code
This is an external example. Check the [repository](https://github.com/pablovela5620/sam2-depthanything) for more information.

You can try the example on HuggingFace space [here](https://huggingface.co/spaces/pablovela5620/sam2-depthanything).

It is highly recommended to run this example locally by cloning the above repo and running (make sure you have [Pixi](https://pixi.sh/latest/#installation) installed):
```
git clone https://github.com/pablovela5620/sam2-depthanything.git
pixi run app
```

0 comments on commit 3731b26

Please sign in to comment.