HELP NEEDED - WANT TO CONTRIBUTE?

[davidpagnon]

Dear all,

I am now doing a relatively unrelated post-doc, and am working on Pose2Sim only on the side. There are tons of features that I think could be useful, and that I cannot find time to implement. If anyone wants to contribute, I'll be more than happy to oblige!

The list is at the end of the Readme file, but I'll put it here for the sake of clarity.

MAIN PROJECTS:
☑ Integrate pose estimation
☑ Integrate scaling and inverse kinematics
☑ Synchronization
☑ Multiple persons triangulation
☑ Blender visualizer add-on
☑ Batch processing
☑ Integrate into Sports2D
▢ Graphical User Interface (see this branch) and/or WebApp.
▢ Real-time triangulation and analysis from connected cameras (see this branch)
▢ Rig skeleton and mesh animation in Blender (see this Discord channel)
▢ Integrate monocular 3D kinematics with RTMPoseW3D (see this Discord channel)
▢ Self-calibration based on keypoint detection (see this Discord channel)
▢ Calibration of moving cameras
▢ Better hand and shoulder OpenSim model (see this Discord channel)
▢ Bundle adjustment for calibration
▢ Video tutorials, documentation on github.io.

I have a clear idea of how to do each of these, so if you are not sure of the details or are not sure whether you can use or develop the skills, I can definitely tell you more about it! Here is an invite to a Discord server if you are interested in discussing it (no commitment at this stage 😛).

If you want to provide edits to the code, please please follow the instructions on the comment below: #40 (comment).

[davidpagnon]

DETAILED TO-DO LIST:

Stuff will be added as needs arise, and removed as they are met.
Full to-do + got-done list there.

GUI:
▢ Simple tkinter interface (or Kivy if we want something nice and portable, or Google Colab, maybe Streamlit). Maybe have a look at the Deeplabcut GUI for inspiration. See Sports2D Demo for OpenPose and Python package installation on Google Drive).
Simply load, edit, and save Config.toml files. Only display relevant options (e.g., do not display options for calculating calibration if conversion is chosen).
▢ App or webapp (e.g., with Napari ).
▢ Pose2Sim_Blender: Rig skeleton (see Caliscope)

Tutorials and Documentation:
▢ Make video tutorials.
▢ Multiple pages for documentation would be clearer that a huge single page. Use Sphinx, MkDocs, or (maybe better) github.io.

Pose:
▢ Optionally let the user select the person of interest in single_person mode: multiperson = true # true, or 'single_auto', or 'single_click'. 'single_auto' selects the person with lowest reprojection error, and 'single_click' lets the user manually select the person of interest.
▢ Implement RTMPoseW3D and monocular 3D kinematics
▢ Directly reading from DeepLabCut .csv or .h5 files instead of converting to .json (triangulation, person association, calibration, synchronization...)
▢ GUI help for DeepLabCut model creation.
▢ Support MMPose, SLEAP, etc.

Calibration:
▢ Currently, clicking the last point confirms the whole selection and switches to the next camera, which can be annoying in case of missclick. Once the reprojection is displayed, show a pop-up message and ask if the user is satisfied. If yes, switch to the next camera; if No, do the clicking again.
▢ Enable calibration with vertical checkerboard.
▢ Calibration of moving cameras: Once object points have been detected or clicked once, track them. Propose to click again when they are lost. Alternatively, use DVPO (see its implementation in GVHMR)
▢ Calibrate cameras by pairs and compute average extrinsic calibration with aniposelib.
▢ Alternatively, self-calibrate with keypoints. Set world reference frame in the end.
▢ Calculate calibration with points rather than board. (1) SBA calibration with wand (cf Argus, see converter here). Set world reference frame in the end.

Synchronization:
▢ Multi-person mode: click on the person to synchronize on

Triangulation:
▢ Solve limb swapping. Try triangulating with opposite side if reprojection error too large (favored approach). Alternatively, ignore right and left sides, use RANSAC or SDS triangulation, and then choose right or left by majority voting. More confidence can be given to cameras whose plane is the most coplanar to the right/left line. See here.
▢ Undistort 2D points before triangulating (and distort them before computing reprojection error). See draft L468 in triangulation.py.
▢ Pre-compile weighted_triangulation and reprojection with @jit(nopython=True, parallel=True) for faster execution.

Filtering:
▢ Implement SmoothNet

OpenSim:
▢ Track hands and face, and add OpenSim articulated hand.
▢ Implement optimal fixed-interval Kalman smoothing for inverse kinematics (this OpenSim fork), or Biorbd)

Others:
▢ Integrate Sports2D for OpenSim analysis from a single camera.
▢ Real-time: Run Pose estimation, Person association, Triangulation, Kalman filter, IK frame by frame (instead of running each step for all frames)
▢ Conda package
▢ Docker image
▢ Run from command line via click or typer
▢ Utilities: Export other data from c3d files into .mot or .sto files (angles, powers, forces, moments, GRF, EMG...)

---

Others (x2)

Pose estimation:

• Animal: It seems like animal pose works on MMPose with animal alias, but not on Pose2Sim. Check if it works with RTMpose onnx file within RTMlib custom mode
• Performance mode: lots of false positives (tripod detected as a human being). Propose pull request to RTMlib so that the performance mode uses another pose detector.
• RTMPose is apparently faster when run from MMPose than from RTMlib, even with the same device and backend. Quantify and understand it

Calibration:

• It seems like manual calibration does not work on MacOS. I do not have such a computer so I cannot debug it, unfortunately
• Before proceeding to the next camera, it would be good to provide a confirmation window, otherwise one misclick may may mess it all up. See this issue
• Calibration based on keypoints would definitely be extremely helpful. @meero, feel free to ask for help if you need it!

Others:

• If we want to be able to run Pose2Sim in real time all the way to the kinematics phase, we need to be able to chose a previously scaled model. To be implemented.
• We have loads of graphic libraries: MMPose for the synchronization, calibration, and plots, CTkinter for the GUI, PyOpenGL for MStudio, Blender for the add-on... And Alex proposes another nice one, Rerun.io. It's a lot of work but it would be worth uniformizing it, at some point.

If anyone feels motivated to check any of these, feel absolutely free to tell us and do so!

[hunminkim98]

Hi, I'm an avid user of pose2sim.
I look forward to new updates every day thanks to your amazing work.
I would like to be of help to you, but I am not a very good developer, so I think I can help you with things like making tutorial videos.
I'm really excited to see your to-do list!

[carlosedubarreto]

thats a great todo list.
Pos2Sim was essential to make MPP2SOS to work, and I would like to give something back, helping to improve the tool.
The main problem I have now is also lack of time.

Some huge things happened in my life (like I had to move 4 times in less than 2 years and because of that and some other things I have basically no emergency funds, so I'm having to work my ass off to pay the debts)

the problem is that getting rid of those problems will take months.
but I'll keep following this topic, to be able to do something as soon as the moment comes.

thanks a lot for the project and for trying to make it even better.

I'm sure, with the help of others, it will go a long way and also help a lot of people!!!

[ANaaim]

Might be able to help :) As suggested above, your pipeline help our lab setting our marker-less pipeline so would be good to help you too. Not sure if i would be the best for the core of the code, however as a good an end user I might to some stuff on tutorial or element for easing the use of the tool-box.

[davidpagnon]

Hi everyone, I'm pleasantly surprised that anyone responded at all, thank you!

@rlagnsals, A tutorial would be great!
Other things that do not require extensive coding skills would be to try and make the code bug, and tell me when the error message is not clear so that I can raise a specific message.
I haven't looked much into it, but hosting the documentation to github.io may not be too complicated. Not sure about this though.

@carlosedubarreto, I don't know the details but your life does sound crazy, and I'm amazed at the way you're involved in so many other projects already! Good luck with everything, and let me know if things calm down a little.

@ANaaim That would be awesome! I feel like a relatively important and easy step would be to make a GUI. It does not have to be beautiful, I'm just thinking of a tkinter window with the exact same info as in the Config.toml file, but presented with tabs for each steps, and drop-down menus for each parameter. I guess it would be nice to be able to import or export a Config.toml file directly from the interface, too.
And I also see that you suggested changing skeletons.py to skeletons.toml, in order to access them more easily. If you are up for doing this, I would also greatly appreciate it!

[hunminkim98]

@davidpagnon I'm eager to contribute to your project and bring value to Pose2Sim!
I'm currently in the process of creating a tutorial video. Additionally, I've been dedicating my efforts to code camera calibration using OpenPose keypoints for contribute. However, I've faced challenges during the joint optimization process. Initially, it diverged instead of converging and subsequently got stuck in a local optimum. This was unexpected, especially since I adhered to the methodology presented in the paper.😱

BY the way, could you guide me on how I can specifically communicate with you and collaboratively work on various projects to make meaningful contributions?

[davidpagnon]

@rlagnsals This would be awesome!
I put "calibration on OpenPose keypoints" low on the list of priorities because it would take me too long to dive in the subject, but if you already started, it could go back up on the bucket list!

Github is a good place for people to talk about how one would like to contribute, but you're right that once that step is passed, we should use another platform to exchange more in-depth. I don't know, I'm on all platforms (WhatsApp, Signal, Telegram, and others), but Teams or Discord may be best. I'm afraid I don't have the skills or time to set up a Discord server, so unless someone wants to do it, Teams may be the most convenient.

Here is an invite for people who may be interested in contributing: https://teams.live.com/l/community/FEAWpdmXJDNT5TA3AI
Edit: moved away from Teams to Discord: https://discord.gg/RpUeuhFN

[davidpagnon]

@carlosedubarreto @rlagnsals @ANaaim
I told you I would keep you informed when it is done, there is now a new version that allows for automatic batch processing :)
More information about it here: https://github.com/perfanalytics/pose2sim?tab=readme-ov-file#batch-processing

Don't hesitate to tell me if there is anything you think should be done differently!
Next project will be (whenever I find time) to handle limb swapping, and to look more in depth into the calibration based on keypoints that @rlagnsals has been working on. And then all the rest :D

[davidpagnon]

Blender add-on now released!

Pose2Sim.Blender.mp4

[timungereth]

Dear @davidpagnon thank you for your amazing work. I just found your work some weeks ago and noticed that we were on the exact same path and that you already did a lot of stuff we planned to do. I think we can help to further develop pose2sim, probably starting with a GUI since we want to start using markerless motion capture within a clinic.
Please do not hesitate to send me an email, so we could kick this off (tim.unger@llui.org). I will have a student working on this from next week on.
Best, Tim

[davidpagnon]

Hi Tim,
Thank you for your appreciation and your help!
Some people from the Université de Valenciennes are also into improving the user-experience: they are starting with conducting interviews with non-specialists (clinicians and coaches), and planned to build a GUI accordingly. It would be best not to make both of your works redundant, but there is probably space for other people working on it!
If you desire, here is a link to the Discord discussion about it: https://discord.com/invite/4mXUdSFjmt

[timungereth]

Hey David, that sounds like a good plan. I however cannot access the discrod server it says the link is not valid. I am not sure whether the problem is on my side or whether there something wrong with the link. Could you please check and share it again, thank you! :)