Inconsistency between optical flow and depth/pose/camera intrinsics

[mmbannert]

Hi,

We discovered a problem with the optic flow, depth, and/or camera pose data that Kubric produces. The problem along with the code to reproduce it is visualized on here.

We are trying to decompose the optical flow from Kubric videos generated with movi_def_worker.py into rigid and nonrigid optical flow (roughly similar to http://arxiv.org/abs/1805.09806). This decomposition requires that we combine camera pose & intrinsics as well as depth information to compute the optical flow. We noticed, however, that the computation of optical flow based on those data is inconsistent with the ground-truth (GT) optical flow that is provided by Kubric. To check that our analysis is indeed correct, we applied it also to Sintel, Monkaa, and Spring as a sanity check. Here, our computation of optical flow is indeed consistent with the ground-truth optical flow that they provide.

Could the inconsistency between reported GT optical flow and depth/pose-based optical flow indicate that something is off about Kubric's GT optical flow, depth, or pose information? Alternatively, are we perhaps misinterpreting some of the data in Kubric, or using them incorrectly?

Help would be greatly appreciated!

Best,
Michael

[mmbannert]

To everyone facing the same questions: I made a PR that includes the kind of documentation that I would have found helpful back then. #307