Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V3 RGBD Node #1198

Open
wants to merge 95 commits into
base: v3_develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
5c579c9
rgb node init impl
Serafadam Dec 6, 2024
111386d
add python bindings for rgbd
Serafadam Dec 9, 2024
b37d9d9
Merge branch 'v3_develop' into v3_rgbd_node
Serafadam Dec 9, 2024
90a9ee6
fix bindings
Serafadam Dec 9, 2024
858d4ba
remove byte padding for point3frgb to fix issues with visualizer
Serafadam Dec 11, 2024
a2d0158
add internal imagealign
Serafadam Dec 11, 2024
6be50ea
add compute
Serafadam Dec 12, 2024
9da4916
update shaders compilation
Serafadam Dec 12, 2024
97fe9fb
add glsllang-tools to ci
Serafadam Dec 13, 2024
f4494be
Merge branch 'v3_develop' into v3_rgbd_node
Serafadam Dec 13, 2024
d7be197
pimpl in rgbd
Serafadam Dec 13, 2024
60bbe9f
update pimpl
Serafadam Dec 13, 2024
0a4a933
fix basalt
Serafadam Dec 18, 2024
b8f9ba4
remove kompute from default deps
Serafadam Dec 18, 2024
86be142
remove internal imagealign
Serafadam Dec 18, 2024
b015ea1
remove gpu code from visualizer example
Serafadam Dec 18, 2024
594343e
remove glslang tools from ci
Serafadam Dec 18, 2024
1ccc2ee
fix bindings
Serafadam Dec 18, 2024
899e8cf
remove align from header
Serafadam Dec 18, 2024
d1083b9
fix examples
Serafadam Dec 19, 2024
0eba35b
add test
Serafadam Dec 19, 2024
1297fa8
Merge branch 'v3_develop' into v3_rgbd_node
Serafadam Dec 19, 2024
d2f45f7
add autocreate test
Serafadam Dec 19, 2024
dd46354
add docstrings
Serafadam Dec 19, 2024
d3ef82b
fix docstrings
Serafadam Dec 19, 2024
fa935ef
try compiling shaders
Serafadam Dec 19, 2024
a9878ba
use blocking get
Serafadam Dec 19, 2024
893f3c1
better performance on cpuMt
Serafadam Dec 20, 2024
9d5fda8
debug arm
Serafadam Dec 20, 2024
b69ca75
kompute support off
Serafadam Dec 30, 2024
9e2965e
try printing logs for kompute
Serafadam Jan 7, 2025
812a004
install additional x deps for arm
Serafadam Jan 7, 2025
74babb4
try debugging pkgconfig
Serafadam Jan 7, 2025
2b2ee64
install xcb-devel
Serafadam Jan 7, 2025
6a449d7
more logs
Serafadam Jan 7, 2025
1cbe1ee
check pkgconfig
Serafadam Jan 8, 2025
391a6ed
pkg config path
Serafadam Jan 8, 2025
538e03a
remove cmake install from arm
Serafadam Jan 8, 2025
8da3fd6
set pkgconfig path
Serafadam Jan 8, 2025
0578e27
set pkgconfig executable directly
Serafadam Jan 8, 2025
de4e727
try setting via other way
Serafadam Jan 8, 2025
72d519c
debug cleanup
Serafadam Jan 9, 2025
83366f7
Merge branch 'v3_develop' into v3_rgbd_node
Serafadam Jan 9, 2025
9e0a58b
update stereo properties in examples, refactor initialization method
Serafadam Jan 9, 2025
641df1c
pr updates
Serafadam Jan 13, 2025
8576003
update vcpkg logs path for build_docstrings
Serafadam Jan 13, 2025
a6b4b47
add python example tests
Serafadam Jan 13, 2025
fe17fe7
fix typo
Serafadam Jan 13, 2025
b2e3d6f
update env vars for eventsmanager
Serafadam Jan 13, 2025
78d697f
fix ci, make compute optional
Serafadam Jan 15, 2025
b11585f
try updating tests
Serafadam Jan 15, 2025
312b7a0
fix python tests
Serafadam Jan 16, 2025
8907adf
update stereodepth build docs
Serafadam Jan 16, 2025
0174f43
update python tests, open3dcpu
Serafadam Jan 16, 2025
7126b7a
explicit align node for rvc4
Serafadam Jan 16, 2025
0776b3a
Merge branch 'v3_develop' into v3_rgbd_node
Serafadam Jan 20, 2025
6caac1f
fix rgbd example size typo
Serafadam Jan 21, 2025
a2cee33
fix examples for rvc4
Serafadam Jan 21, 2025
f82f1cc
Merge branch 'v3_develop' into v3_rgbd_node
Serafadam Jan 23, 2025
97fd833
add getstereopair for oak4d
Serafadam Jan 24, 2025
26fe959
update stereo autocreate to work on rvc4
Serafadam Jan 24, 2025
e8248ae
Merge branch 'v3_develop' into v3_rgbd_node
Serafadam Jan 24, 2025
c75906f
Merge branch 'v3_develop' into v3_rgbd_node
Serafadam Jan 27, 2025
be89d47
bump fw
Serafadam Jan 27, 2025
b9019e6
Merge branch 'v3_develop' into v3_rgbd_node
Serafadam Jan 29, 2025
e2c0152
get intrinsics from frame directly
Serafadam Jan 29, 2025
e89dacb
merge v3_develop
Serafadam Feb 3, 2025
d21bcde
add apt update to pytest ci
Serafadam Feb 3, 2025
4c5d238
Merge branch 'v3_develop' into v3_rgbd_node
Feb 4, 2025
704383f
Update the examples to 640x400
Feb 4, 2025
67b36ff
update sizes in some examples
Serafadam Feb 5, 2025
a7d4f93
install python rerun o3d deps for example tests
Serafadam Feb 5, 2025
dec8f33
rgbdata serialization
Serafadam Feb 5, 2025
649feda
rgbdata bindings
Serafadam Feb 5, 2025
1529130
PR updates
Serafadam Feb 5, 2025
f2bc564
test for rgbddata
Serafadam Feb 5, 2025
5ed3f37
update sizes in visualizer example
Serafadam Feb 13, 2025
1d505b9
merge v3_develop
Serafadam Feb 18, 2025
b3cab8f
running sync on host temporarily
Serafadam Feb 18, 2025
e8e5c01
tag rerun 0.16.0
Serafadam Feb 18, 2025
ca83360
pr updates
Serafadam Feb 19, 2025
dbd2d8e
revert sync subnode changes
Serafadam Feb 19, 2025
df115b1
remove unused method from example
Serafadam Feb 19, 2025
491d62f
add comments to visualizer examples
Serafadam Feb 19, 2025
61f08a1
Merge branch 'v3_develop' into v3_rgbd_node
Serafadam Feb 20, 2025
1ae7d07
update examples script
Serafadam Feb 21, 2025
22f8cb4
skip o3d example on win
Serafadam Feb 24, 2025
95cb2da
update rgbd example
Serafadam Feb 24, 2025
890ae53
don't run rgbd_py on windows ci
Serafadam Feb 25, 2025
2676562
Merge branch 'v3_develop' into v3_rgbd_node
Serafadam Feb 26, 2025
c8029f0
remove rerun install from python win examples
Serafadam Feb 27, 2025
902b8c8
Merge branch 'v3_develop' into v3_rgbd_node
Serafadam Feb 28, 2025
5f92bf5
visualizer update
Serafadam Feb 28, 2025
d4f0ac9
update to use point3rgba
Serafadam Feb 28, 2025
cd4037b
add requesting undistorted output on rvc2
Serafadam Feb 28, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update pimpl
  • Loading branch information
Serafadam committed Dec 13, 2024
commit 60bbe9f241412f61522d1e2b4670a07e0b3da1ae
10 changes: 5 additions & 5 deletions examples/cpp/HostNodes/rgbd.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ class RerunNode : public dai::NodeCRTP<dai::node::ThreadedHostNode, RerunNode> {
rec.log_static("world", rerun::ViewCoordinates::FLU);
rec.log("world/ground", rerun::Boxes3D::from_half_sizes({{3.f, 3.f, 0.00001f}}));
while(isRunning()) {
auto pclIn = inputPCL.tryGet<dai::PointCloudData>();
auto pclIn = inputPCL.get<dai::PointCloudData>();
if(pclIn != nullptr) {
std::vector<rerun::Position3D> points;
std::vector<rerun::Color> colors;
Expand All @@ -40,8 +40,6 @@ int main() {
dai::Pipeline pipeline;
// Define sources and outputs
int fps = 30;
int width = 640;
int height = 400;
// Define sources and outputs
auto left = pipeline.create<dai::node::MonoCamera>();
auto right = pipeline.create<dai::node::MonoCamera>();
Expand All @@ -65,9 +63,11 @@ int main() {
stereo->setRectifyEdgeFillColor(0); // black, to better see the cutout
stereo->enableDistortionCorrection(true);
stereo->initialConfig.setLeftRightCheckThreshold(10);
stereo->setDepthAlign(dai::StereoDepthProperties::DepthAlign::CENTER);
rgbd->setOutputMeters(true);
rgbd->setGPUDevice(1);
rgbd->useGPU();

auto *out = color->requestOutput(std::pair<int, int>(1280, 720));
auto *out = color->requestOutput(std::pair<int, int>(1280, 720), dai::ImgFrame::Type::RGB888i);
left->out.link(stereo->left);
right->out.link(stereo->right);

Expand Down
7 changes: 4 additions & 3 deletions examples/cpp/Visualizer/visualizer_rgbd.cpp
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's wait on the needed Visualizer changes so this examples start working.

So either that blocks that PR or we add the example later.

Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,6 @@ int main() {
dai::Pipeline pipeline;
// Define sources and outputs
int fps = 30;
int width = 640;
int height = 400;
// Define sources and outputs
auto left = pipeline.create<dai::node::MonoCamera>();
auto right = pipeline.create<dai::node::MonoCamera>();
Expand All @@ -50,7 +48,9 @@ int main() {
stereo->enableDistortionCorrection(true);
stereo->initialConfig.setLeftRightCheckThreshold(10);

auto *out = color->requestOutput(std::pair<int, int>(1280, 720));
rgbd->setGPUDevice(1);
rgbd->useGPU();
auto *out = color->requestOutput(std::pair<int, int>(1280, 720), dai::ImgFrame::Type::RGB888i);
left->out.link(stereo->left);
right->out.link(stereo->right);

Expand All @@ -59,6 +59,7 @@ int main() {

remoteConnector.addTopic("pcl", rgbd->pcl);
pipeline.start();
remoteConnector.registerPipeline(pipeline);
auto device = pipeline.getDefaultDevice();
device->setIrLaserDotProjectorIntensity(0.7);
// Main loop
Expand Down
1 change: 1 addition & 0 deletions include/depthai/pipeline/node/host/RGBD.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ class RGBD : public NodeCRTP<ThreadedHostNode, RGBD> {
void useCpuMt();
void useGPU();
void setGPUDevice(uint32_t deviceIndex);
void setCPUThreadNum(uint32_t numThreads);
void printDevices();
private:
class Impl;
Expand Down
17 changes: 2 additions & 15 deletions shaders/rgbd2pointcloud.comp
Original file line number Diff line number Diff line change
Expand Up @@ -6,25 +6,16 @@ layout(std430, binding = 0) buffer DepthBuffer {
float depth[];
};

// Binding 1: RGB input (float array, assume each pixel has 3 floats: R,G,B)
layout(std430, binding = 1) buffer RGBBuffer {
float rgb[];
};

// Binding 2: Intrinsics buffer [fx, fy, cx, cy, scale, width, height]
layout(std430, binding = 2) buffer IntrinsicsBuffer {
layout(std430, binding = 1) buffer IntrinsicsBuffer {
float intrinsics[];
};

// Binding 3: Output XYZ buffer (float array, 3 floats per pixel)
layout(std430, binding = 3) buffer XYZBuffer {
layout(std430, binding = 2) buffer XYZBuffer {
float xyz[];
};

// Binding 4: Output RGB buffer (float array, 3 floats per pixel)
layout(std430, binding = 4) buffer OutRGBBuffer {
float outRgb[];
};

void main() {
uint i = gl_GlobalInvocationID.x;
Expand Down Expand Up @@ -52,9 +43,5 @@ void main() {
xyz[i * 3 + 1] = y;
xyz[i * 3 + 2] = z;

// Copy RGB directly
outRgb[i * 3 + 0] = rgb[i * 3 + 0];
outRgb[i * 3 + 1] = rgb[i * 3 + 1];
outRgb[i * 3 + 2] = rgb[i * 3 + 2];
}

15 changes: 5 additions & 10 deletions src/pipeline/datatype/PointCloudData.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -40,21 +40,16 @@ std::vector<Point3fRGB> PointCloudData::getPointsRGB() {
void PointCloudData::setPoints(const std::vector<Point3f>& points) {
auto size = points.size();
std::vector<uint8_t> data(size * sizeof(Point3f));
auto* dataPtr = reinterpret_cast<Point3f*>(data.data());
for(unsigned int i = 0; i < size; i++) {
dataPtr[i] = points[i];
}
setData(data);
std::memcpy(data.data(), points.data(), size * sizeof(Point3f));
setData(std::move(data));
setColor(false);
}

void PointCloudData::setPointsRGB(const std::vector<Point3fRGB>& points) {
auto size = points.size();
std::vector<uint8_t> data(size * sizeof(Point3fRGB));
auto* dataPtr = reinterpret_cast<Point3fRGB*>(data.data());
for(unsigned int i = 0; i < size; i++) {
dataPtr[i] = points[i];
}
setData(data);
std::memcpy(data.data(), points.data(), size * sizeof(Point3fRGB));
setData(std::move(data));
setColor(true);
}

Expand Down
Loading