Pose2Sim Hand

[tlsgil2012-design]

Hello. I have a question for the Pose2Sim development team.
Currently, when using Pose2Sim's Whole-body model, I understand it is based on the COCO 133 keypoint model. This model uses only two keypoints per hand, totaling four keypoints, in addition to the torso and foot keypoints.
However, Hand Pose Estimation typically tracks 21 keypoints per hand, totaling 42 keypoints for both hands.
I'm curious if there is a specific reason why only two hand keypoints are used in the 3D modeling process, even though all 133 keypoints can be extracted from the 2D image.
Additionally, I would like to ask if it is possible to add 21 keypoints per hand, totaling 42 hand keypoints, to the body + feet keypoints to achieve more detailed motion analysis.
I look forward to your response. Thank you.

[davidpagnon]

Hi,

You can choose among two models in Config.toml to have wrist movement: pose_model = 'Whole_body', or 'Whole_body_wrist'. My answer will be different if you are only interested in the triangulation() stage (trc files with marker trajectories), or the kinematics() one (scaled model and joint angle time series).

• In the first case: choosing "Whole_body" will definitely triangulate all finger points, which is what you want. Whole_body_wrist is interesting because focusing on only 2 points on the hand is enough for wrist motion, it creates shorter files, and makes triangulation faster.

• In the second case: If you want to go all the way to kinematic optimization, this is a bit more complicated. The .osim models I use do not have any finger joints, so I cannot use all points anyway. That said, it is entirely possible to combine a hand model such as this one with the original Pose2Sim one. I don't have time to do it right now, but if you want to give it a go I'm happy to help you!

[tlsgil2012-design]

Thanks for the quick reply. I want to apply it all the way to kinematic optimization. Could you help me with combining the models using the link you shared?

[davidpagnon]

I don't have much time but I can guide you through :)
Is there any way you can join the Discord server? It would make chatting easier.