Feature/extract stereo filters #1308

lnotspotl · 2025-04-28T06:28:00Z

Purpose

This PR adds stereo-depth and other depth-confidence TOF filters as two nodes. The two nodes are implemented as host-runnable device nodes. By default, the two nodes run on host. An important feature the two nodes have is that all filter settings can be modified at runtime.

Clickup task: https://app.clickup.com/t/86c0x6wq6

Screencast.from.2025-04-28.01-24-50.webm

Testing script (RVC2)

#!/usr/bin/env python3

import cv2
import depthai as dai
import numpy as np

extended_disparity = False
subpixel = True
lr_check = True

# Create pipeline
with dai.Pipeline() as pipeline:
    # Define sources and outputs
    monoLeft = pipeline.create(dai.node.MonoCamera)
    monoRight = pipeline.create(dai.node.MonoCamera)
    depth = pipeline.create(dai.node.StereoDepth)
    filterPipeline = pipeline.create(dai.node.SequentialDepthFilters)

    filterPipeline.addFilter(dai.node.SequentialDepthFilters.MedianFilterParams())
    filterPipeline.addFilter(dai.node.SequentialDepthFilters.SpeckleFilterParams())
    filterPipeline.addFilter(dai.node.SequentialDepthFilters.TemporalFilterParams())
    filterPipeline.addFilter(dai.node.SequentialDepthFilters.SpatialFilterParams())

    print("Setting filter pipeline to run on host")
    filterPipeline.setRunOnHost(True)

    # Properties
    monoLeft.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
    # monoLeft.setCamera("left")
    monoRight.setBoardSocket(dai.CameraBoardSocket.CAM_B)
    monoRight.setResolution(dai.MonoCameraProperties.SensorResolution.THE_400_P)
    # monoRight.setCamera("right")
    monoRight.setBoardSocket(dai.CameraBoardSocket.CAM_C)

    # Create a node that will produce the depth map (using disparity output as it's easier to visualize depth this way)
    depth.setLeftRightCheck(lr_check)
    depth.setExtendedDisparity(extended_disparity)
    depth.setSubpixel(subpixel)
    depth.inputConfig.setBlocking(False)
    configQueue = depth.inputConfig.createInputQueue()

    configInputQueue = filterPipeline.config.createInputQueue()

    # Linking
    monoLeft.out.link(depth.left)
    monoRight.out.link(depth.right)
    depthQueue = depth.disparity.createOutputQueue()

    depth.disparity.link(filterPipeline.input)
    filterOutputQueue = filterPipeline.output.createOutputQueue()

    threshold = 1
    pipeline.start()
    import time
    t_switch = time.time()
    enabled = True
    while pipeline.isRunning():
        inDisparity : dai.ImgFrame = depthQueue.get() # blocking call, will wait until a new data has arrived
        frame = inDisparity.getFrame()
        filterFrame = filterOutputQueue.get()
        filterFrame = (filterFrame.getFrame() * (255 / depth.initialConfig.getMaxDisparity())).astype(np.uint8)
        frame = (frame * (255 / depth.initialConfig.getMaxDisparity())).astype(np.uint8)
        cv2.imshow("disparity", frame)
        frame = cv2.applyColorMap(frame, cv2.COLORMAP_JET)
        cv2.imshow("disparity_color", frame)
        cv2.imshow("filtered_disparity_color", cv2.applyColorMap(filterFrame, cv2.COLORMAP_JET))
        if time.time() - t_switch > 1:
            print(f"Filter enabled: {enabled}")
            t_switch = time.time()
            config = dai.SequentialDepthFiltersConfig()
            config.filterIndex = 0
            config.filterParams.enable = enabled
            config.filterParams.median = dai.StereoDepthConfig.MedianFilter.KERNEL_3x3
            configInputQueue.send(config)
            enabled = not enabled

        def update():
            print(f"Updating to {threshold}")
            message = dai.StereoDepthConfig()
            message.setConfidenceThreshold(threshold)
            configQueue.send(message)

        key = cv2.waitKey(1)
        if key == ord('q'):
            break
        if key == ord('j'):
            threshold += 1
            update()
        if key == ord('k'):
            threshold -= 1
            if threshold < 1: threshold = 1
            update()

moratom

Thanks Jakub!

Left some comments&suggestions, generally looks good.
Let's also add an example (can be similar to stereo_depth_from_host.py) and tests before merging.

moratom · 2025-04-28T12:37:59Z

src/pipeline/datatype/ImgFrame.cpp

@@ -187,6 +187,23 @@ ImgFrame& ImgFrame::setMetadata(const std::shared_ptr<ImgFrame>& sourceFrame) {
    return setMetadata(*sourceFrame);
 }

+ImgFrame& ImgFrame::setDataFrom(const ImgFrame& sourceFrame) {


I think we should go with copyDataFrom to make it clearer it's not a move.

include/depthai/pipeline/node/host/DepthFilters.hpp

moratom · 2025-04-28T13:20:44Z

include/depthai/properties/DepthFiltersProperties.hpp

+typedef dai::StereoDepthConfig::PostProcessing::SpatialFilter SpatialFilterParams;
+typedef dai::StereoDepthConfig::PostProcessing::SpeckleFilter SpeckleFilterParams;
+typedef dai::StereoDepthConfig::PostProcessing::TemporalFilter TemporalFilterParams;
+typedef std::variant<MedianFilterParams, SpatialFilterParams, SpeckleFilterParams, TemporalFilterParams> FilterParams;


Probbably better to move to the filters to a common location and then use using ... in both StereoDepth and here

moratom · 2025-04-28T13:30:25Z

include/depthai/pipeline/datatype/DepthFiltersConfig.hpp

+    /**
+     * Index of the filter to be applied
+     */
+    std::int32_t filterIndex;
+
+    /**
+     * Parameters of the filter to be applied
+     */
+    FilterParams filterParams;
+


Any reason why we didn't go with std::vector here?

The idea is as follows: Say we have filters, applied sequentially, (F1 | F2 | F3 | F4 | F5). When one wants to modify the properties any one filter, say F3, one can simply specify its index and properties without modifying or having to worry about all the other filters. This will throw an error if the param's type does not match the expected type of the filter at the given index.

moratom · 2025-04-28T13:31:25Z

include/depthai/properties/DepthFiltersProperties.hpp

+struct SequentialDepthFiltersProperties : PropertiesSerializable<Properties, SequentialDepthFiltersProperties> {
+    /**
+     * List of filters (the type of which is determined by the filter parameters) to apply to the input frame
+     */
+    std::vector<FilterParams> filters;
+};
+


Let's rather use an initialConfig, to not duplicate parameters.

Read this, the properties and the config are different, we have a fixed sequence of filters, each of which one can set the properties at runtime for.

moratom · 2025-04-28T13:32:44Z

include/depthai/properties/DepthFiltersProperties.hpp

+struct DepthConfidenceFilterProperties : PropertiesSerializable<Properties, DepthConfidenceFilterProperties> {
+    /**
+     * Threshold for the confidence filter
+     */
+    float confidenceThreshold = 0.0f;
+};


Same as for DepthFilter properties

src/pipeline/node/host/DepthFilters.cpp

SzabolcsGergely · 2025-04-30T21:40:28Z

src/pipeline/node/host/DepthFilters.cpp

+
+constexpr const size_t PERSISTENCY_LUT_SIZE = 256;
+
+struct TemporalFilterParams {


why not break out each implementation in their own .cpp ?

SzabolcsGergely · 2025-04-30T21:44:22Z

src/pipeline/node/host/DepthFilters.cpp

@@ -0,0 +1,904 @@
+#include "depthai/pipeline/node/host/DepthFilters.hpp"


i'd rather call these ImageFilters, they can be used for general purposes, if one wants to.
Only depthConfidence filter is ToF specific?
E.g. temporal filter could be used on input images to stereo depth to reduce noise. Same for median.

Correct, the DepthConfidenceFilter is specifically designed to used alongside the ToF node.

…epthai-core into feature/extract_stereo_filters

Copilot

Pull Request Overview

Adds two new host-runnable filter nodes (image filters and depth-confidence filter) to the pipeline, along with their data/config types, properties, bindings, and example scripts.

Introduce ImgFrame::copyDataFrom and clone methods to duplicate frames
Define ImageFilters and DepthConfidenceFilter nodes, their configs, properties, and Python/JS bindings
Refactor StereoDepthConfig to use centralized FilterParams, update datatype enums, CMake, and examples

Reviewed Changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/pipeline/datatype/ImgFrame.cpp	Implement `copyDataFrom` and `clone` for frame duplication
include/depthai/pipeline/datatype/ImgFrame.hpp	Declare `copyDataFrom`/`clone` and update doc comments
src/pipeline/datatype/DatatypeEnum.cpp	Add new datatypes to hierarchy for image/depth-confidence
include/depthai/pipeline/datatype/DatatypeEnum.hpp	Extend `DatatypeEnum` with new config entries
include/depthai/properties/ImageFiltersProperties.hpp	Define properties for both filter nodes
include/depthai/pipeline/node/ImageFilters.hpp	Create `ImageFilters` and `DepthConfidenceFilter` nodes
include/depthai/pipeline/datatype/StereoDepthConfig.hpp	Refactor post-processing filters to `FilterParams`
include/depthai/pipeline/FilterParams.hpp	Introduce centralized filter parameter structs/enums
include/depthai/pipeline/datatype/ImageFiltersConfig.hpp	New buffer messages for runtime filter reconfiguration
bindings/python/.../FilterParamsBindings.*	Python bindings for filter parameters
bindings/python/.../ImageFiltersBindings.cpp	Python/node bindings for new filter nodes
bindings/js/bindings.cpp	Add `ImageFiltersConfig` to JS bindings
CMakeLists.txt	Link OpenCV calib3d, include new source and binding files
examples/python/.../stereo_depth_filters.py	Python example for image filters on stereo disparity
examples/python/RVC2/ToF/tof_host_filter.py	Python example for depth-confidence filter with ToF node

Comments suppressed due to low confidence (1)

src/pipeline/datatype/ImgFrame.cpp:200

New clone method should have unit tests verifying deep copy of both metadata and pixel data.

std::shared_ptr<ImgFrame> ImgFrame::clone() const {

Copilot · 2025-05-25T00:35:43Z

include/depthai/pipeline/datatype/ImgFrame.hpp

@@ -310,6 +310,25 @@ class ImgFrame : public Buffer, public ProtoSerializable {
     */
    ImgFrame& setMetadata(const std::shared_ptr<ImgFrame>& sourceFrame);

+    /**
+     * Convience function to set the data of the ImgFrame


Typo in doc comment: 'Convience' should be 'Convenience'.

Copilot · 2025-05-25T00:35:44Z

include/depthai/pipeline/datatype/ImgFrame.hpp

+    ImgFrame& copyDataFrom(const ImgFrame& sourceFrame);
+
+    /**
+     * Convience function to set the data of the ImgFrame


Typo in doc comment: 'Convience' should be 'Convenience'.

Copilot · 2025-05-25T00:35:44Z

src/pipeline/datatype/ImgFrame.cpp

+    std::vector<uint8_t> data(sourceFrame.data->getData().begin(), sourceFrame.data->getData().end());
+    setData(std::move(data));


[nitpick] This copies the frame data via iterator range; consider using the underlying buffer or a single setData(sourceFrame.data->getData()) call to avoid redundant allocations.

Suggested change

std::vector<uint8_t> data(sourceFrame.data->getData().begin(), sourceFrame.data->getData().end());

setData(std::move(data));

setData(sourceFrame.data->getData());

…[skip ci]

…Camera node [skip ci]

lnotspotl added 4 commits April 28, 2025 00:38

Initial depth filter host node implementation

b102b01

Enable dynamic filter property setting

9d02266

Dynamic depth confidence node config

d45e14c

Dynamic depth confidence node config

6e44920

lnotspotl self-assigned this Apr 28, 2025

lnotspotl requested a review from moratom April 28, 2025 06:28

moratom reviewed Apr 28, 2025

View reviewed changes

SzabolcsGergely reviewed Apr 30, 2025

View reviewed changes

lnotspotl added 6 commits May 6, 2025 03:54

Update naming, fix ToF filter

373b2a7

Move depth filters out of host node folder

af39e67

follow camelCase naming

2d18856

Merge branch 'v3_develop' into feature/extract_stereo_filters

9e29e10

Update variable names

3dfc5f9

remove unused variables

d8983c9

lnotspotl marked this pull request as draft May 6, 2025 08:51

lnotspotl added 9 commits May 6, 2025 04:58

follow camelCase naming convention

e236d5b

follow camelCase naming convention

b89c59c

Rename DepthFilters to ImageFilters

a0dbb4f

extract commonalities between image filters and stereo depth node

c1ec546

Update docstrings

1b53532

Update docstrings

79c4e51

add filter host node example

22648d5

fix temporal filter for different ImgFrame types

13435b0

Remove redundant printf

81c3436

lnotspotl requested review from SzabolcsGergely and moratom May 6, 2025 13:07

lnotspotl marked this pull request as ready for review May 6, 2025 14:43

lnotspotl added 3 commits May 7, 2025 04:39

fix python docs generation

8357c13

fix docs build

ad0e426

Update ImageFilters.cpp

dbccf99

lnotspotl added 6 commits May 11, 2025 17:52

add tof host filters example

a977d56

Merge branch 'feature/extract_stereo_filters' of github.com:luxonis/d…

8e28914

…epthai-core into feature/extract_stereo_filters

Update ImageFilters.cpp

bd1d73c

use smart pointers, use depthai checks instead of throws

29d69e9

fix confidence

3f1384a

fix confidence

c7f23b2

SzabolcsGergely requested a review from Copilot May 25, 2025 00:34

Copilot AI reviewed May 25, 2025

View reviewed changes

lnotspotl added 4 commits June 9, 2025 09:22

merge v3_develop in [skip ci]

02bbfa0

Fix errors, rename DepthConfidenceFilter to ToFDepthConfidenceFilter …

00fafff

…[skip ci]

add preset modes, fix warnings and errors [skip ci]

38d2598

update examples to use the Camera node instead of the depracated Mono…

97525dd

…Camera node [skip ci]


		constexpr const size_t PERSISTENCY_LUT_SIZE = 256;

		struct TemporalFilterParams {

		@@ -0,0 +1,904 @@
		#include "depthai/pipeline/node/host/DepthFilters.hpp"

		std::vector<uint8_t> data(sourceFrame.data->getData().begin(), sourceFrame.data->getData().end());
		setData(std::move(data));

	std::vector<uint8_t> data(sourceFrame.data->getData().begin(), sourceFrame.data->getData().end());
	setData(std::move(data));
	setData(sourceFrame.data->getData());

Feature/extract stereo filters #1308

Are you sure you want to change the base?

Feature/extract stereo filters #1308

Uh oh!

Conversation

lnotspotl commented Apr 28, 2025

Purpose

Testing script (RVC2)

Uh oh!

moratom left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!