How can I de-project from capture_depth_float_buffer to 3D points in the point cloud?

[jackjansen]

I have a point cloud (captured by some RGBD cameras, but I guess that doesn't matter). I have code to interactively select markers on that point cloud by using VisualizerWithEditing(), picking the corners manually and then using get_picked_points() to get the 3D coordinates.

I also have code to use capture_screen_float_buffer() to grab a screen shot, feed that into ArucoDetector() and get the four corners of my marker as 2D image coordinates.

But I cannot find a way to deproject these 2D image coordinates back to the 3D points that correspond to them. The code must be in there somewhere, because obviously picking points with left click in VisualizerWithEditing() needs it...

Is there a way to do what I want?

Or, alternatively, if there is another solution for turning the point cloud into a 2D image with a way to deproject 2D image coordinates back to 3D coordinates that would be great too.