-
-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add RSSDK 2.0 (librealsense2) grabber #2214
Add RSSDK 2.0 (librealsense2) grabber #2214
Conversation
io/CMakeLists.txt
Outdated
@@ -5,9 +5,9 @@ set(SUBSYS_DEPS common octree) | |||
set(build TRUE) | |||
PCL_SUBSYS_OPTION(build "${SUBSYS_NAME}" "${SUBSYS_DESC}" ON) | |||
if(WIN32) | |||
PCL_SUBSYS_DEPEND(build "${SUBSYS_NAME}" DEPS ${SUBSYS_DEPS} OPT_DEPS openni openni2 ensenso davidSDK dssdk rssdk pcap png vtk) | |||
PCL_SUBSYS_DEPEND(build "${SUBSYS_NAME}" DEPS ${SUBSYS_DEPS} OPT_DEPS openni openni2 ensenso davidSDK dssdk rssdk librealsense pcap png vtk) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm assuming that since librealsense is cross platform it should be added to both of these lists
Patrick and Daniel thanks for contributing this! :) We're still in C++03 and the migration to C++14 is bound to only happen after our next release, meaning this is somewhat low priority for now. I hope that doesn't conflict too much with your "agenda". |
Thanks sergio - since we are working from PCL+librealsense source it's working fine for us. There is no rush to merge on our side. Just let us know when the C++ 11 migration is live, and we'll bring this branch up to date with PCL master |
class PCL_EXPORTS RealSense2Grabber : public pcl::Grabber | ||
{ | ||
public: | ||
RealSense2Grabber(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it would be good so that to receive the serial number in constructor to open specific devices or multiple devices.
public:
- RealSense2Grabber ();
+ RealSense2Grabber (const std::string& serial_number = "");
protected:
+ std::string serial_number;
- RealSense2Grabber::RealSense2Grabber ()
+ RealSense2Grabber::RealSense2Grabber (const std::string& serial_number)
: running (false)
, quit (false)
+ , serial_number (serial_number)
void
RealSense2Grabber::start ()
{
running = true;
rs2::config cfg;
+ if (!serial_number.empty())
+ cfg.enable_device (serial_number);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can now start the grabber with a specific device
io/src/real_sense_2_grabber.cpp
Outdated
auto uvptr = points.get_texture_coordinates(); | ||
|
||
uint8_t r, g, b; | ||
for (auto& p : cloud->points) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please consider to parallelizing using OpenMP.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've added OpenMP to the loops. Please let me know if I'm using the standard PCL OpenMP conventions.
io/src/real_sense_2_grabber.cpp
Outdated
|
||
cfg.enable_stream(RS2_STREAM_COLOR, 424, 240, RS2_FORMAT_RGB8, 30); | ||
cfg.enable_stream(RS2_STREAM_DEPTH, 424, 240, RS2_FORMAT_ANY, 30); | ||
cfg.enable_stream(RS2_STREAM_INFRARED, 424, 240, RS2_FORMAT_ANY, 30); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that it should provide some way to set resolution and frame rate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cmake/Modules/FindLibRealSense.cmake
Outdated
|
||
# Libraries | ||
find_library(LIBREALSENSE_LIBRARY NAMES librealsense.lib) | ||
find_library(LIBREALSENSE_LIBRARY_DEBUG NAMES librealsense.lib) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the librealsense2 library name is realsense2.lib
.
Why is it librealsense2.lib
?
And, Please add suffix (e.g. _d
) for debug library name. (realsense2_d.lib
)
@SergioRAgostinho Where is used C++11 features? |
cmake/Modules/FindLibRealSense.cmake
Outdated
find_package(PkgConfig QUIET) | ||
|
||
# Include directories | ||
find_path(LIBREALSENSE_INCLUDE_DIR librealsense2/rs.h) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please consider to add these options (HINTS
, PATHS
, and PATH__SUFFIXES
) to find_path()
and find_library()
.
I think it can write as following.
(But, I'm not familiar with Linux, and MacOS. It need more comments from other reviewer.)
find_path(LIBREALSENSE_INCLUDE_DIR librealsense2/rs.h
PATHS "$ENV{realsense2_DIR}"
"$ENV{PROGRAMW6432}/librealsense2" # for self built library
"$ENV{PROGRAMFILES}/librealsense2" # for self built library
"$ENV{PROGRAMFILES}/Intel RealSense SDK 2.0" # for pre built library
# "Please specify search paths for Linux and MacOS"
PATH_SUFFIXES include)
set(LIBREALSENSE_INCLUDE_DIRS ${LIBREALSENSE_INCLUDE_DIR})
mark_as_advanced(LIBREALSENSE_INCLUDE_DIRS)
set(LIBREALSENSE_RELEASE_NAME realsense2)
set(LIBREALSENSE_DEBUG_NAME realsense2_d)
set(LIBREALSENSE_SUFFIX x86)
if(CMAKE_SIZEOF_VOID_P EQUAL 8)
set(LIBREALSENSE_SUFFIX x64)
endif()
find_library(LIBREALSENSE_LIBRARY
NAMES ${LIBREALSENSE_RELEASE_NAME}
PATHS "$ENV{realsense2_DIR}"
"$ENV{PROGRAMW6432}/librealsense2"
"$ENV{PROGRAMFILES}/librealsense2"
"$ENV{PROGRAMFILES}/Intel RealSense SDK 2.0"
# "Please specify search paths for Linux and MacOS"
PATH_SUFFIXES lib lib/${LIBREALSENSE_SUFFIX})
find_library(LIBREALSENSE_LIBRARY_DEBUG
NAMES ${LIBREALSENSE_DEBUG_NAME}
PATHS "$ENV{realsense2_DIR}"
"$ENV{PROGRAMW6432}/librealsense2"
"$ENV{PROGRAMFILES}/librealsense2"
"$ENV{PROGRAMFILES}/Intel RealSense SDK 2.0"
# "Please specify search paths for Linux and MacOS"
PATH_SUFFIXES lib lib/${LIBREALSENSE_SUFFIX})
if(NOT LIBREALSENSE_LIBRARY_DEBUG)
set(LIBREALSENSE_LIBRARY_DEBUG ${LIBREALSENSE_LIBRARY})
endif()
set(LIBREALSENSE_LIBRARIES optimized ${LIBREALSENSE_LIBRARY} debug ${LIBREALSENSE_LIBRARY_DEBUG})
mark_as_advanced(LIBREALSENSE_LIBRARY LIBREALSENSE_LIBRARY_DEBUG)
@daniel-packard Thank you for your contribution. |
https://github.com/IntelRealSense/librealsense/releases/tag/v2.8.0 In the actual code Guys, some really really quick (review) comments:
|
io/src/real_sense_2_grabber.cpp
Outdated
return 30.0f; | ||
} | ||
|
||
pcl::PointCloud<pcl::PointXYZ>::Ptr RealSense2Grabber::convertDepthToPointXYZ(const rs2::points & points) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not implemented yet?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is now implemented
cmake/Modules/FindLibRealSense.cmake
Outdated
|
||
# Libraries | ||
set(LIBREALSENSE_RELEASE_NAME realsense2) | ||
set(LIBREALSENSE_DEBUG_NAME realsense2_d) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Of course, you are right about the name of the libraries (librealsense.lib vs realsense2) 👍
I also liked your recommended update to include PATHS and PATH_SUFFIXES in the calls to find_path and find_library, so I've updated that as well. I don't know what appropriate values would be on linux/mac either -- hopefully a Linux/Mac user can comment with additional recommendations for those platforms
@SergioRAgostinho I've implemented some of the C++11 changes you mentioned regarding std::thread and std::mutex. You mentioned help in setting up unit tests for the grabber. Yeah that would definitely be helpful but we don't want the scope of this pull request to get too big. |
@daniel-packard I'd like to adjust coding style according to PCL.
- int32_t i;
+ int32_t i;
- function();
+ function ();
- int32_t function ();
+ int32_t
+ function ();
- int i;
+ int32_t i; |
What I mentioned was setting up boilerplate code to exemplify how to mock a grabber. The unit tests and the actual mocking behavior will need to be written by you guys because you have the hardware so actually know how to properly mock it. Providing unit tests is well within the scope of this PR.
I don't usually request this unless there's some compelling argument behind it. I do normally advise to be considerate about your variable types based on the prior information i.e.:
@UnaNancyOwen any specific motives behind this request I'm failing to see? |
.gitignore
Outdated
@@ -0,0 +1 @@ | |||
build/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove this please. Add it to your .git/info/exclude file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You just taught me a new git trick - thanks!
And it is removed.
@SergioRAgostinho I was intended for this. Sorry, my request expression was not appropriate. |
It is an first idea for support capture from bag file. class PCL_EXPORTS RealSense2Grabber : public pcl::Grabber
{
public:
RealSense2Grabber (const uint32_t serial_number = 0);
RealSense2Grabber (const std::string& file_name);
protected:
uint32_t serial_number;
std::string file_name;
} // capture from sensor
RealSense2Grabber::RealSense2Grabber (const uint32_t serial_number = 0)
: serial_number (serial_number)
{
}
// capture from file
RealSense2Grabber (const std::string& file_name)
: file_name (file_name)
{
}
void
RealSense2Grabber::start ()
{
running = true;
rs2::config cfg;
// capture from file
if (file_name.rfind (".bag") == 4)
{
cfg.enable_device_from_file (file_name);
}
// capture from sensor
else
{
if (serial_number)
cfg.enable_device (std::to_string (serial_number));
cfg.enable_stream (RS2_STREAM_COLOR, deviceWidth, deviceHeight, RS2_FORMAT_RGB8, targetFps);
cfg.enable_stream (RS2_STREAM_DEPTH, deviceWidth, deviceHeight, RS2_FORMAT_ANY, targetFps);
cfg.enable_stream (RS2_STREAM_INFRARED, deviceWidth, deviceHeight, RS2_FORMAT_ANY, targetFps);
}
pipe.start(cfg);
thread = std::thread(&RealSense2Grabber::threadFunction, this);
} |
A quick comment that it is actually @patrickabadi who has contributed the C++ code, so all praise/criticism should be directed towards him 😁 That said, we are happy to reformat according to the PCL style guide Have you considered adding a .editorconfig file to the repo to help facilitate some of the auto-formatter settings? (e.g. https://gist.github.com/buoto/7e2cff6c17c108658fc751df845b1040)
Ah, I misunderstood what was being offered/requested -- yes please, your help setting this up would be very appreciated, and we can provide unit tests (although we might require some guidance, or pointers other implementations within PCL that would provide a good model )
Isn't a bag file a concept from ROS, or has PCL adopted it too? In general, I think the ability to support record/playback is excellent -- and if this is the recommended way, we will work together to add support Last note -- we are amazed at the engagement from you guys, and the excellent feedback 💯 We will have to start spreading out our contributions a little bit so we can continue working on our day jobs too. What is the approximate timeframe for adding C++ 11 support so we can make our plans accordingly? |
librealsense2 have feature that capture from bag files. It is easy and simple. rs2::config config;
config.enable_device_from_file ("file.bag");
rs2::pipeline pipeline;
pipeline.start (config); |
@UnaNancyOwen I updated the code to match the pcl style guide. Please let me know if everything looks good. I also added the bag file loading capability. Note: We've been testing RealSense2Grabber on the pcl in_hand_scanner sample and so far it was been working great. |
/** \brief Constructor | ||
* \param[in] serial_number optional to start up a specific device | ||
*/ | ||
RealSense2Grabber (const std::string& serial_number = "", const std::string& file_name = ""); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this implementation, It need to pass two arguments when capturing from file.
I think that this specification is generally not favored.
// capture from bag file
boost::shared_ptr<pcl::RealSense2Grabber> grabber = boost::make_shared<pcl::RealSense2Grabber> ("", "file.bag");
Is there a special reason for this implementation?
I think it should define two contractors. (#2214 (comment))
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @UnaNancyOwen that makes sense since the user can only do one or the other. I've separated the constructors now.
cmake/Modules/FindLibRealSense.cmake
Outdated
"$ENV{PROGRAMFILES}/librealsense2" # for self built library | ||
"$ENV{PROGRAMFILES}/Intel RealSense SDK 2.0" # for pre built library | ||
"$ENV{LIBREALSENSE_ROOT}" | ||
# "Please specify search paths for Linux and MacOS" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should it add "/usr/local/"
and "/usr/"
for Linux and MacOS?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems reasonable, and looking at libusb, these seem to be some of the standard hints on linux: https://github.com/PointCloudLibrary/pcl/blob/46cb8fe5589e88e36d79f9b8b8e5f4ff4fceb5de/cmake/Modules/Findlibusb-1.0.cmake
I'll do a quick review of the librealsense docs and see if they install to any of these paths by default on linux/mac
I have RealSense D415, D435, and D435i. I will test with them, and test with ros-bag file on Windows. |
PCLConfig.cmake.in
Outdated
@@ -216,6 +216,19 @@ macro(find_rssdk) | |||
find_package(RSSDK) | |||
endmacro() | |||
|
|||
macro(find_rssdk2) | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove extra blank line in first and last of this macro.
*/ | ||
|
||
#ifndef PCL_IO_REAL_SENSE_2_GRABBER_H | ||
#define PCL_IO_REAL_SENSE_2_GRABBER_H |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We replaced all include-guards to #pragma once
in #2617.
Please replace this include-guard.
- #ifndef PCL_IO_REAL_SENSE_2_GRABBER_H
- #define PCL_IO_REAL_SENSE_2_GRABBER_H
+ #pragma once
...
- #endif /* PCL_IO_REAL_SENSE_2_GRABBER_H */
I have confirmed that this grabber works correctly with sensor and file. 👍 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a few inline comments, plus one bigger question. Could you please explain why the critical sections are where they are? E.g. why mutex needs to be locked while busy-waiting for frames?
PCLConfig.cmake.in
Outdated
elseif(NOT realsense2_DIR) | ||
get_filename_component(realsense2_DIR "@REALSENSE2_INCLUDE_DIRS@" PATH) | ||
set(realsense2_DIR "${realsense2_DIR}/lib/cmake/realsense2" CACHE PATH "The directory containing realsense2Config.cmake") | ||
endif(PCL_ALL_IN_ONE_INSTALLER) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an outdated CMake syntax to repeat the if
condition in endif
and in the meantime we got rid of this everywhere in PCL. Please remove here as well.
PCLConfig.cmake.in
Outdated
set(realsense2_DIR "${realsense2_DIR}/lib/cmake/realsense2" CACHE PATH "The directory containing realsense2Config.cmake") | ||
endif(PCL_ALL_IN_ONE_INSTALLER) | ||
find_package(RSSDK2) | ||
endmacro(find_rssdk2) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same here.
cfg.enable_stream (RS2_STREAM_DEPTH, device_width_, device_height_, RS2_FORMAT_ANY, target_fps_); | ||
cfg.enable_stream (RS2_STREAM_INFRARED, device_width_, device_height_, RS2_FORMAT_ANY, target_fps_); | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, I'm still in favor of the last Tsukasa's proposal.
I just looked at it and I think it might be old code, there's no reason to lock while getting data. I'll take it out |
io/src/real_sense_2_grabber.cpp
Outdated
{ | ||
sw.reset (); | ||
|
||
rs2::depth_frame depth = nullptr; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This fails to compile with GCC 7.4 on Ubuntu.
error: conversion from ‘std::nullptr_t’ to non-scalar type ‘rs2::depth_frame’ requested
Which makes sense, how can you assign nullptr
to a class?
I've added librealsense to our Ubuntu 16.04 image that is used on CI. From now on we can see whether the new grabber can be built on that platform. I've also restarted the previous run, so you can see the error I poster before. |
I can successfully build the grabber now. However, when I try to visualize point clouds from the grabber, I get:
I'm using D415 and |
I haven't had a chance to try in linux. That error comes from librealsense. Are you able to use the D415 with the "RealSense Viewer" that is provided by librealsense? Will try to spin up an Ubuntu VM to test |
Yep, I tried with their viewer and it worked. Also different combinations of resolution/fps. |
Sorry to bring this from the dead. I would be interested in releasing the working Windows version of this for this release, even if under some |
Fine for me. I don't think we need to use "experimental" directory, it's sufficient to mention this explicitly in the documentation. |
OK. My plan was to compile and include these files only under Windows and fork a new a issue to address the failures under Linux. |
|
||
if (signal_PointXYZRGB->num_slots () > 0 || signal_PointXYZRGBA->num_slots () > 0) | ||
{ | ||
assert ( prof.get_stream ( RS2_STREAM_COLOR ).format () == RS2_FORMAT_RGB8 ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This check is irrelevant because this condition is already tested in line 129. In case it is false
an exception is thrown.
assert ( prof.get_stream ( RS2_STREAM_COLOR ).format () == RS2_FORMAT_RGB8 ); | ||
} | ||
|
||
assert ( prof.get_stream ( RS2_STREAM_DEPTH ).format () == RS2_FORMAT_Z16 ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same comment as above, the check is already performed in line 130.
@taketwo, @UnaNancyOwen, @kunaltyagi Do you mind if I merge this and make the corrections in another PR? |
I agree. |
Thanks everyone! 🎉 |
Hi PCL team - @patrickabadi has implemented a librealsense 2.0 based
RealSense2Grabber
, and we'd like to make a contribution. I noticed the last comment on #1633 implies @UnaNancyOwen was planning to implement a librealsense 2.0 grabber, but maybe this will provide a good starting point that covers a lot of the boilerplate cmake configuration.quick notes: