Skip to content

Commit

Permalink
Update rgb-depth aligned example to be more generic; support alpha sc…
Browse files Browse the repository at this point in the history
…aling parameter
  • Loading branch information
SzabolcsGergely committed Jul 5, 2023
1 parent e5573f0 commit 0efec25
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 11 deletions.
2 changes: 1 addition & 1 deletion depthai-core
Submodule depthai-core updated 0 files
29 changes: 19 additions & 10 deletions examples/StereoDepth/rgb_depth_aligned.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,16 @@
import cv2
import numpy as np
import depthai as dai
import argparse

# Weights to use when blending depth/rgb image (should equal 1.0)
rgbWeight = 0.4
depthWeight = 0.6

parser = argparse.ArgumentParser()
parser.add_argument('-alpha', type=float, default=None, help="Alpha scaling parameter to increase float. [0,1] valid interval.")
args = parser.parse_args()
alpha = args.alpha

def updateBlendWeights(percent_rgb):
"""
Expand All @@ -21,9 +26,6 @@ def updateBlendWeights(percent_rgb):
depthWeight = 1.0 - rgbWeight


# Optional. If set (True), the ColorCamera is downscaled from 1080p to 720p.
# Otherwise (False), the aligned depth is automatically upscaled to 1080p
downscaleColor = True
fps = 30
# The disparity is computed at this resolution, then upscaled to RGB resolution
monoResolution = dai.MonoCameraProperties.SensorResolution.THE_720_P
Expand All @@ -34,7 +36,7 @@ def updateBlendWeights(percent_rgb):
queueNames = []

# Define sources and outputs
camRgb = pipeline.create(dai.node.ColorCamera)
camRgb = pipeline.create(dai.node.Camera)
left = pipeline.create(dai.node.MonoCamera)
right = pipeline.create(dai.node.MonoCamera)
stereo = pipeline.create(dai.node.StereoDepth)
Expand All @@ -48,15 +50,17 @@ def updateBlendWeights(percent_rgb):
queueNames.append("disp")

#Properties
camRgb.setBoardSocket(dai.CameraBoardSocket.CAM_A)
camRgb.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
rgbCamSocket = dai.CameraBoardSocket.CAM_A

camRgb.setBoardSocket(rgbCamSocket)
camRgb.setSize(1280, 720)
camRgb.setFps(fps)
if downscaleColor: camRgb.setIspScale(2, 3)

# For now, RGB needs fixed focus to properly align with depth.
# This value was used during calibration
try:
calibData = device.readCalibration2()
lensPosition = calibData.getLensPosition(dai.CameraBoardSocket.CAM_A)
lensPosition = calibData.getLensPosition(rgbCamSocket)
if lensPosition:
camRgb.initialControl.setManualFocus(lensPosition)
except:
Expand All @@ -71,14 +75,19 @@ def updateBlendWeights(percent_rgb):
stereo.setDefaultProfilePreset(dai.node.StereoDepth.PresetMode.HIGH_DENSITY)
# LR-check is required for depth alignment
stereo.setLeftRightCheck(True)
stereo.setDepthAlign(dai.CameraBoardSocket.CAM_A)
stereo.setDepthAlign(rgbCamSocket)

# Linking
camRgb.isp.link(rgbOut.input)
camRgb.video.link(rgbOut.input)
left.out.link(stereo.left)
right.out.link(stereo.right)
stereo.disparity.link(disparityOut.input)

camRgb.setMeshSource(dai.CameraProperties.WarpMeshSource.CALIBRATION)
if alpha is not None:
camRgb.setCalibrationAlpha(alpha)
stereo.setAlphaScaling(alpha)

# Connect to device and start pipeline
with device:
device.startPipeline(pipeline)
Expand Down

8 comments on commit 0efec25

@Erol444
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@szabi-luxonis I found 2 issues using latest develop commit (d1c103e8):

  1. Stereo doesn't seem to be correctly scaled to color frame. I get correct color resolution (1280x720, as set), but depth map gets upscaled to 1920x1080:
print('RGB shape', frameRgb.shape, "Disparity shape", frameDisp.shape)
# adding this line to 138 returns:
RGB shape (720, 1280, 3) Disparity shape (1080, 1920, 3)
  1. After downscaling disparity to 720x1280 and visualizing the result, it's not aligned correctly:
    image
    This one looks better, but still not ideally aligned:
    image

@SzabolcsGergely
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Erol444

Would you mind describing the steps to reproduce?
Does latest release work?

@Erol444
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@szabi-luxonis
Here's gist for repro - essentially the same script as above, just resizing disp frame).
No, it also doesn't work with latest 2.22:
image
image

@SzabolcsGergely
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Erol444

It works with OV9782.
image

IMX378 errors out:

[18443010A19C381300] [1.8] [1.065] [Camera(0)] [error] Not possible to create warp params. Error: WARP_SWCH_ERR_CACHE_TO_SMALL 

[18443010A19C381300] [1.8] [1.241] [StereoDepth(3)] [error] Disparity/depth width must be multiple of 16, but RGB camera width is 1352. Set output size explicitly using 'setOutputSize(width, height)'.

Tested with cc5370b66205d36fae7b772ec6abf0dc6284fb24, when it was introduced.

@Erol444
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@szabi-luxonis I see, with OV9782 it works ok in my example as well (latest develop), but doesn't work well with imx378 (still works tho).

@SzabolcsGergely
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Erol444 that means that RGB alignment itself works, but there are other issues. Either with the Camera node and/or with how intrinsics are calculated. Or other issue. For me latest develop doesn't run.
I assume configuring ColorCamera at 1920x1080 then undistorting/dewarping w/ ImageManip would work correctly.

@SzabolcsGergely
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think latest commit broke it, which added 1352x... resolution to IMX378

@zemsozeri
Copy link

@zemsozeri zemsozeri commented on 0efec25 Oct 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, I tried to use latest code but I got following error on OAK-D:

blended = cv2.addWeighted(frameRgb, rgbWeight, frameDisp, depthWeight, 0)
cv2.error: OpenCV(4.8.1) /Users/xperience/GHA-OpenCV-Python/_work/opencv-python/opencv-python/opencv/modules/core/src/arithm.cpp:650: error: (-209:Sizes of input arguments do not match) The operation is neither 'array op array' (where arrays have the same size and the same number of channels), nor 'array op scalar', nor 'scalar op array' in function 'arithm_op'

Reason is that latestPacket["disp"] is None

Please sign in to comment.