Make backprojected point cloud from depth image individually selectable

[genemerewether]

Is it possible to make points from a backprojected point cloud (from depth map) individually selectable? Would be nice to have the same ability for selections of those points to project to (other) cameras, for example.

[jleibs]

This is something we've talked about quite a bit. The short answer is we definitely want to make this possible eventually, but it poses some surprising structural complexities.

The main issue is actually in how we think of instances within a batch and how that relates to images (and by extension depth images and their associated backprojected points).

These issues have some more context and would likely allow exactly what you're looking for:

• Treat tensor values as instances #1756
• Selecting a point (instance) and selecting a pixel is conceptually different when it shouldn't #853