Reproducing reported results with TotalCapture Data

[colincqian]

Hello Xinyu,

As I show in the attached image (reported results in brackets), I am trying to reproduce the reported results from the paper with the provided pre-trained model.

The test on DIP data matches the reported results well, however, the testing results on TotalCapture Data are far away from the reported ones. The dataset I am using follows your comment and README, where I get TotalCapture with IMU from the DIP website [ORIGINAL TotalCapture DATA W/ CORRESPONDING REFERENCE SMPL Poses], and the gt data from Total Capture Website.

I am not sure why the Testing on TotalCapture Data has such a large gap. Could u help me figure it out?

[Xinyu-Yi]

You can download the processed TotalCapture data here for evaluation: https://cloud.tsinghua.edu.cn/f/847a75f10ed244adb16d/?dl=1

This data seem different from [ORIGINAL TotalCapture DATA W/ CORRESPONDING REFERENCE SMPL Poses], but it can reproduce the paper's results.

[colincqian]

Hello Xinyu, I found a small mismatch with the preprocess.py code for AMASS data. The ji_mask = torch.tensor([18, 19, 4, 5, 15, 0]) means that it uses "lelbow ,relbow" to synthesize IMU measurement. However, the real-world IMU placement is on "lwrist, rwrist". Is that a typo in the code or do you do that for some other reasons?

[Xinyu-Yi]

Thank you. That dataset works! I have already get permission from DIP and ToTalCapture data. Could you provide me with the source dataset? That will be very helpful. Here is my e-mail address chqian@student.ethz.ch.

This dataset is generated from the original TotalCapture dataset and the smpl pose that DIP sent me some years ago. I am sorry that it is difficult to find the original dataset. You can directly extract the smpl pose from the dataset. All other data can be generated from official TotalCapture dataset.

[Xinyu-Yi]

Hello Xinyu, I found a small mismatch with the preprocess.py code for AMASS data. The ji_mask = torch.tensor([18, 19, 4, 5, 15, 0]) means that it uses "lelbow ,relbow" to synthesize IMU measurement. However, the real-world IMU placement is on "lwrist, rwrist". Is that a typo in the code or do you do that for some other reasons?

This is because we need the global rotation of the elbow. but not the wrist. Actually we place the IMU on the forearm near the wrist. It still measures the orientation of the elbow.

[colincqian]

Thank you so much for answering. One last question about finetuning (Finetuning, problem reproducing results for TotalCapture Xinyu-Yi/TransPose#42) I have a similar problem as it is mentioned in this issue, where metrics on DIP-IMU are good, but metrics on Total Capture is around 25(SIP)/7.5(Joint Position). I am wondering what is your setup for the fine-tuning process. ( e.g. learning rate/ epochs/loss function/)

[Xinyu-Yi]

I remember that I just finetuned the network with DIP-IMU dataset for several epochs using a 10-times lower lr. If you do not stop the finetuning and let the finetuning converge, maybe the network will overfit to DIP-IMU, leading to a low error on DIP-IMU but high error on TotalCapture