Creating new "ground truth" for several datasets

[isarandi]

Hi, thanks for this amazing work.

Do you have any plans for running your method on other datasets and releasing the resulting poses? This would be very beneficial for correcting many ground truth errors. Specifically I'm thinking of MPI-INF-3DHP (some annotations are wrong) and HumanEva-I (in some sequences the head ground truth is wrong), in addition the already mentioned H3.6M (problems with S9) and CMU-Panoptic (ground truth unavailable for many sequences, e.g. dance + errors).

I think your results could be better than the original ground truth in many cases. So by e.g. leave-one-subject-out training and testing, one could generate new, polished "ground truth" for each subject of a particular dataset (to avoid memorizing the training set errors).

[karfly]

Hey, thank you for your interest.

Next step for us is to add CMU Panoptic dataset support. Then we can think about adding other multi-view datasets.

We have some vague plans about annotating/reannotating datasets. Maybe the community can help us with it? 😊

[dulibubai]

If I want to train the model with CMU Panoptic dataset ,that means I'd better modify the dataset prepare codes based on Human3.6M precesssing way? Can you add CMU Panoptic dataset support? Can you add CMU Panoptic dataset more details in your paper?Thanks a lot!

[karfly]

@dulibubai
We’re going to add CMU Panoptic dataset support soon, but if you need it right now, you can implement it yourself using Human3.6M as a reference.

What exact details of CMU Panoptic dataset training are you interested in?

[dulibubai]

When I download CMU dataset in its official network, I found most of the sequences have not labels,only some have , and there many people in most of them ,just single human is several..So when you train with it,which sequences you chose before,and how to split the train,val and test dataset when you train the model with CMU dataset? Thanks again.

[karfly]

@dulibubai
We use train/val splits provided by authors of the original paper "Monocular Total Capture: Posing Face, Body, and Hands in the Wild".

Each scene contains multiple recorded persons => for each person an interval is provided in format[start_frame, end_frame]. Here is the list of scene names split into train/val:

train:
    - 171026_pose3
      - [1000, 3000]

    - 171026_pose2
      - [1000, 7500]
      - [8000, 14000]

    - 171026_pose1
      - [380, 7300]
      - [7900, 14500]
      - [15400, 22400]

    - 171204_pose4
      - [500, 4300]
      - [4900, 8800]
      - [9400, 13200]
      - [14200, 17800]
      - [18700, 22500]
      - [23050, 27050]
      - [28000, 31600]

    - 171204_pose3
      - [500, 4400]
      - [5400, 9000]

    - 171204_pose2
      - [350, 4300]
      - [5000, 8800]
      - [9600, 13600]
      - [14300, 18500]
      - [19600, 23500]
      - [24200, 28200]
      - [28800, 32800]
      - [33500, 37700]

    - 171204_pose1
      - [300, 4100]
      - [4800, 8900]
      - [10000, 13600]
      - [14000, 18200]
      - [18500, 22900]
      - [23500, 27600]

  val:
    - 171204_pose5
      - [400, 4300]
      - [5000,  8500]
      - [9500, 13400]
      - [14200, 18000]
      - [19000, 22600]
      - [23500, 27100]

    - 171204_pose6
      - [1000, 4500]
      - [5150, 9100]
      - [9830, 13800]
      - [14370, 18300]
      - [19000, 22900]

[dulibubai]

Thanks for your sharing sincerely! As show in CMU dataset, every sequences have 31 cameras,and how to split 31 cameras images for train and val dataset? Thanks again.

[karfly]

@dulibubai
We used val cameras: ["00_02", "00_13", "00_16", "00_18"]

[dulibubai]

Yeah! Thanks a lot!

[dulibubai]

Hi! I have another question, Is it convenient to provide the 2d bbox lable file in every camera images with CMU that extracted by object detection net?

[karfly]

@dulibubai

I've uploaded our Mask R-CNN detections to the Google Drive.
The format of the detection is the same as in the Human3.6M dataset:

detection == (left, upper, right, lower, confidence)

[dulibubai]

Thanks a lot!

[dulibubai]

@karfly ,
1)In the generate-labels-npy-multiview.py,what's the effect of the square_the_bbox(bbox) function?
def square_the_bbox(bbox):
top, left, bottom, right = bbox
width = right - left
height = bottom - top
if height < width:
center = (top + bottom) * 0.5
top = int(round(center - width * 0.5))
bottom = top + width
else:
center = (left + right) * 0.5
left = int(round(center - height * 0.5))
right = left + height
return top, left, bottom, right
2)In the human36m.py, why do you ' TLBR to LTRB' with bbox information?
bbox = shot['bbox_by_camera_tlbr'][camera_idx][[1,0,3,2]] # TLBR to LTRB
Thanks a lot!

[shrubb]

@dulibubai

1. Object detectors will output rectangluar bounding boxes with arbitrary aspect ratio (this can possibly be true even for ground truth bounding boxes). However, since we are training a CNN, we'd like all input images to be of same size and obviously of same aspect ratio. Therefore, we decided to adjust all bounding boxes to 1:1 height-width ratio, i.e. make them square (we could have chosen some other ratio). This function does this for one box by growing the smallest side.

2. That is nothing important, it is there just for convenience. I think when we were writing human36m.py, some functions would already require LTRB bboxes (like crop_image(), scale_bbox()), so we had to adapt to them.

[dulibubai]

@shrubb ,Thanks a lot!
I have another question to ask for you
In the generate-labels-npy-multiview.py,
1)Why transposes R? Did R in Human3.6M have transposed when stored?
2)Why not store 'T' in camera_retval['t'] , Did 'T' in Human3.6M's cameras_param file not equal true 't'?
camera_retval['R'] = np.array(camera_params['R']).T
camera_retval['t'] = -camera_retval['R'] @ camera_params['T']
3)When I use my externel dataset, I don't need to do so with R's tranpose ?
If I can receive t directly, I can store t in camera_retval['t'] directly?

Thanks very much!

[shrubb]

@dulibubai
In our code, the projection math (for all datasets) is handled by the code in mvn/utils/multiview.py. There, we adopted an OpenCV-like convention regarding camera model-related formulae (I might be wrong here). Human3.6M's intrinsics and extrinsics came in a different format, they used different projection formulae, and a different distortion model. So, the code you quoted simply converts R and T shipped with Human3.6M to our format.