HELP NEEDED - WANT TO CONTRIBUTE?

[davidpagnon]

Dear all,

I am now doing a relatively unrelated post-doc, and am working on Pose2Sim only on the side. There are tons of features that I think could be useful, and that I cannot find time to implement. If anyone wants to contribute, I'll be more than happy to oblige!

The list is at the end of the Readme file, but I'll put it here for the sake of clarity.

MAIN PROJECTS:
☑ Blender visualizer add-on
☑ Batch processing
☑ Multiple persons triangulation
☑ Synchronization
☑ Integrate pose estimation
☑ Integrate scaling and inverse kinematics
▢ Graphical User Interface
▢ Video tutorials, documentation on github.io.
▢ Self-calibration based on keypoint detection
▢ Calibration of moving cameras
▢ Bundle adjustment for calibration
▢ Rig skeleton in Blender
▢ Integrate monocular 3D kinematics with RTMPoseW3D
▢ Calibrate Sports2D data, run IK, and then integrate the toolbox in Pose2Sim

I have a clear idea of how to do each of these, so if you are not sure of the details or are not sure whether you can use or develop the skills, I can definitely tell you more about it! Here is an invite to a Discord server if you are interested in discussing it (no commitment at this stage 😛).

If you want to provide edits to the code, please please follow this guide on how to fork, modify, and push code, and submit a pull request. I would appreciate it if you provided as much useful information as possible about how you modified the code, and a rationale for why you're making this pull request. Please also specify on which operating system and on which Python version you have tested the code.

[davidpagnon]

DETAILED TO-DO LIST:

Stuff will be added as needs arise, and removed as they are met.
Full to-do + got-done list there.

GUI:
▢ Simple tkinter interface (or Kivy if we want something nice and portable, or Google Colab, maybe Streamlit). Maybe have a look at the Deeplabcut GUI for inspiration. See Sports2D Demo for OpenPose and Python package installation on Google Drive).
Simply load, edit, and save Config.toml files. Only display relevant options (e.g., do not display options for calculating calibration if conversion is chosen).
▢ App or webapp (e.g., with Napari ).
▢ Pose2Sim_Blender: Rig skeleton (see Caliscope)

Tutorials and Documentation:
▢ Make video tutorials.
▢ Multiple pages for documentation would be clearer that a huge single page. Use Sphinx, MkDocs, or (maybe better) github.io.

Pose:
▢ Optionally let the user select the person of interest in single_person mode: multiperson = true # true, or 'single_auto', or 'single_click'. 'single_auto' selects the person with lowest reprojection error, and 'single_click' lets the user manually select the person of interest.
▢ Implement RTMPoseW3D and monocular 3D kinematics
▢ Directly reading from DeepLabCut .csv or .h5 files instead of converting to .json (triangulation, person association, calibration, synchronization...)
▢ GUI help for DeepLabCut model creation.
▢ Support MMPose, SLEAP, etc.

Calibration:
▢ Currently, clicking the last point confirms the whole selection and switches to the next camera, which can be annoying in case of missclick. Once the reprojection is displayed, show a pop-up message and ask if the user is satisfied. If yes, switch to the next camera; if No, do the clicking again.
▢ Enable calibration with vertical checkerboard.
▢ Calibration of moving cameras: Once object points have been detected or clicked once, track them. Propose to click again when they are lost.
▢ Calibrate cameras by pairs and compute average extrinsic calibration with aniposelib.
▢ Alternatively, self-calibrate with keypoints. Set world reference frame in the end.
▢ Calculate calibration with points rather than board. (1) SBA calibration with wand (cf Argus, see converter here). Set world reference frame in the end.

Synchronization:
▢ Multi-person mode: click on the person to synchronize on

Triangulation:
▢ Solve limb swapping. Try triangulating with opposite side if reprojection error too large (favored approach). Alternatively, ignore right and left sides, use RANSAC or SDS triangulation, and then choose right or left by majority voting. More confidence can be given to cameras whose plane is the most coplanar to the right/left line. See here.
▢ Undistort 2D points before triangulating (and distort them before computing reprojection error). See draft L468 in triangulation.py.
▢ Pre-compile weighted_triangulation and reprojection with @jit(nopython=True, parallel=True) for faster execution.

Filtering:
▢ Implement SmoothNet

OpenSim:
▢ Track hands and face, and add OpenSim articulated hand.
▢ Implement optimal fixed-interval Kalman smoothing for inverse kinematics (this OpenSim fork), or Biorbd)

Others:
▢ Integrate Sports2D for OpenSim analysis from a single camera.
▢ Real-time: Run Pose estimation, Person association, Triangulation, Kalman filter, IK frame by frame (instead of running each step for all frames)
▢ Conda package
▢ Docker image
▢ Run from command line via click or typer
▢ Utilities: Export other data from c3d files into .mot or .sto files (angles, powers, forces, moments, GRF, EMG...)

[hunminkim98]

Hi, I'm an avid user of pose2sim.
I look forward to new updates every day thanks to your amazing work.
I would like to be of help to you, but I am not a very good developer, so I think I can help you with things like making tutorial videos.
I'm really excited to see your to-do list!

[carlosedubarreto]

thats a great todo list.
Pos2Sim was essential to make MPP2SOS to work, and I would like to give something back, helping to improve the tool.
The main problem I have now is also lack of time.

Some huge things happened in my life (like I had to move 4 times in less than 2 years and because of that and some other things I have basically no emergency funds, so I'm having to work my ass off to pay the debts)

the problem is that getting rid of those problems will take months.
but I'll keep following this topic, to be able to do something as soon as the moment comes.

thanks a lot for the project and for trying to make it even better.

I'm sure, with the help of others, it will go a long way and also help a lot of people!!!

[ANaaim]

Might be able to help :) As suggested above, your pipeline help our lab setting our marker-less pipeline so would be good to help you too. Not sure if i would be the best for the core of the code, however as a good an end user I might to some stuff on tutorial or element for easing the use of the tool-box.

[davidpagnon]

Hi everyone, I'm pleasantly surprised that anyone responded at all, thank you!

@rlagnsals, A tutorial would be great!
Other things that do not require extensive coding skills would be to try and make the code bug, and tell me when the error message is not clear so that I can raise a specific message.
I haven't looked much into it, but hosting the documentation to github.io may not be too complicated. Not sure about this though.

@carlosedubarreto, I don't know the details but your life does sound crazy, and I'm amazed at the way you're involved in so many other projects already! Good luck with everything, and let me know if things calm down a little.

@ANaaim That would be awesome! I feel like a relatively important and easy step would be to make a GUI. It does not have to be beautiful, I'm just thinking of a tkinter window with the exact same info as in the Config.toml file, but presented with tabs for each steps, and drop-down menus for each parameter. I guess it would be nice to be able to import or export a Config.toml file directly from the interface, too.
And I also see that you suggested changing skeletons.py to skeletons.toml, in order to access them more easily. If you are up for doing this, I would also greatly appreciate it!

[hunminkim98]

@davidpagnon I'm eager to contribute to your project and bring value to Pose2Sim!
I'm currently in the process of creating a tutorial video. Additionally, I've been dedicating my efforts to code camera calibration using OpenPose keypoints for contribute. However, I've faced challenges during the joint optimization process. Initially, it diverged instead of converging and subsequently got stuck in a local optimum. This was unexpected, especially since I adhered to the methodology presented in the paper.😱

BY the way, could you guide me on how I can specifically communicate with you and collaboratively work on various projects to make meaningful contributions?

[davidpagnon]

@rlagnsals This would be awesome!
I put "calibration on OpenPose keypoints" low on the list of priorities because it would take me too long to dive in the subject, but if you already started, it could go back up on the bucket list!

Github is a good place for people to talk about how one would like to contribute, but you're right that once that step is passed, we should use another platform to exchange more in-depth. I don't know, I'm on all platforms (WhatsApp, Signal, Telegram, and others), but Teams or Discord may be best. I'm afraid I don't have the skills or time to set up a Discord server, so unless someone wants to do it, Teams may be the most convenient.

Here is an invite for people who may be interested in contributing: https://teams.live.com/l/community/FEAWpdmXJDNT5TA3AI
Edit: moved away from Teams to Discord: https://discord.gg/RpUeuhFN

[davidpagnon]

@carlosedubarreto @rlagnsals @ANaaim
I told you I would keep you informed when it is done, there is now a new version that allows for automatic batch processing :)
More information about it here: https://github.com/perfanalytics/pose2sim?tab=readme-ov-file#batch-processing

Don't hesitate to tell me if there is anything you think should be done differently!
Next project will be (whenever I find time) to handle limb swapping, and to look more in depth into the calibration based on keypoints that @rlagnsals has been working on. And then all the rest :D

[davidpagnon]

Blender add-on now released!

Pose2Sim.Blender.mp4

[timungereth]

Dear @davidpagnon thank you for your amazing work. I just found your work some weeks ago and noticed that we were on the exact same path and that you already did a lot of stuff we planned to do. I think we can help to further develop pose2sim, probably starting with a GUI since we want to start using markerless motion capture within a clinic.
Please do not hesitate to send me an email, so we could kick this off ([email protected]). I will have a student working on this from next week on.
Best, Tim

[davidpagnon]

Hi Tim,
Thank you for your appreciation and your help!
Some people from the Université de Valenciennes are also into improving the user-experience: they are starting with conducting interviews with non-specialists (clinicians and coaches), and planned to build a GUI accordingly. It would be best not to make both of your works redundant, but there is probably space for other people working on it!
If you desire, here is a link to the Discord discussion about it: https://discord.com/invite/4mXUdSFjmt

[davidpagnon]

Multi-person analysis is now supported!

Sorting people across view, and then across frames.
Set [project] multi_person = true for each trial that contains multiple persons.
Set [triangulation] reorder_trc = true if you need to run OpenSim and match the generated .trc files with the static trials.
Make sure that the order of [markerAugmentation] participant_height and participant_mass matches the order of the static trials.

Feel free to test it and tell me if anything does not go as expected.

[davidpagnon]

Multiperson analysis much faster now: as fast as easymocap, and more robust results.
A challenging test scene used to take 1h30 to be processed, it is now under 2 minutes.

[davidpagnon]

Thanks to all the people who have boosted the project lately!

• Automatic synchronization is now available (thanks @hunminkim98 for working on it with me)
• Non-batch processing is possible for quick analysis

[davidpagnon]

OpenSim scaling and inverse kinematics are now fully automatic and integrated into Pose2Sim!
Bonus: no need for a static trial anymore.
Thanks to @peterlololsss, we worked on it together :)

[divan]

Hi, I was keeping an eye on your project for a while, as I need to develop a solution for single-camera motion capture for specific sport with export to Blender. I've been using Wonder Dynamics pipeline so far, with impressive results, but it's a closed proprietary solution, with a lot of extra functionaility that I don't need and don't control. So exploring open-source options.

While the documentation for this repo seems to be great, I'm still not sure if the tools used in this project will allow a single camera motion estimation, including the moving camera (so camera motion estimation is a must in my case). In case it's doable, my first thought was that I would love to have a GUI desktop app for that, and that's where I think I can potentially contribute as I have pretty good experience making cross-platform Flutter apps, including desktop ones.

[Kent-McPhail]

Hi I would like to contribute and currently have quite a bit of time on my hands so point me in a direction and I could definitely get started.

[davidpagnon]

Hi @divan and @Kent-McPhail!

Thanks for your interest, that's awesome! And sorry for the late answer, I'm on paternity leave since last night so I won't be very available for a couple weeks 😊

@divan, right now we have two packages, Sports2D and Pose2Sim, although the first will probably be integrated in the latter eventually. Sports2D works with a single camera, but only for 2D motion. I still need to integrate kinematic optimization. Pose2Sim works with multiple cameras, for any motion. But monocular 3D pose estimation models are getting better, so we will probably offer the possibility to skip calibration and triangulation to go straight to the kinematic optimization part.

I have never used WonderDymanics but I just had a look at this video and it does seem like they are doing a great job! I have never done camera pose estimation but I can see a lot of github projects around it, hopefully not too difficult to integrate. Of course, the company has a lot more human and financial means than an open-source solution, but I think it is achievable :)

Note that for now, the skeletons imported in Blender are not rigged, they are just individual bones. I wonder if this is something you could be interested in?

A flutter app would be awesome, too! @AYLARDJ has been working on a tkinter app lately, on this branch. Unfortunately, I did not find time to review it yet, but he made a demo video of how it could work. Maybe you should coordinate and see how you can work together? Is there a chance for you join the discord server?

batch_for_pose2sim_2.mov

batch_for_pose2sim_1.mov

@Kent-McPhail, when you look at the to-do-list, is there anything that looks close ebough to your skillset and your interest? Or anything else that is not there but that you think could be useful?
We can probavly find a mission from this point on :)

[Kent-McPhail]

Congratulations on the new kid first off.

A couple areas I am interested in honestly.
One is a rigged skeleton in blender as I am currently trying to work with data from openbiomechanics.org where I want to be able to visualize the data in blender. Not much direction in there other than I would like to be able

Utilities: Export other data from c3d files into .mot or .sto files (angles, powers, forces, moments, GRF, EMG...)
This seems like something I could work on as well and a good starting off point too

[davidpagnon]

Great indeed! I'll answer more on the Discord channel

[davidpagnon]

New: It is now possible to use custom detection and pose models:

• Other than 'lightweight', 'balanced', or 'performance' modes, you can use any other pose estimation models through RTMLib (hand, face, animal, or any custom trained models).

• The (optional) detection model, pose model, and input sizes can be written in a dictionary (within triple quotes) as shown below. Models can be local paths or URLs, with .onnx or .zip extensions. Make sure the input_sizes are within square brackets.

  # Equivalent to mode='balanced', with body_with_feet pose model
  mode = """{'det_class':'YOLOX',
         'det_model':'https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/onnx_sdk/yolox_m_8xb8-300e_humanart-c2c7a14a.zip',
         'det_input_size':[640, 640],
         'pose_class':'RTMPose',
         'pose_model':'https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/onnx_sdk/rtmpose-m_simcc-body7_pt-body7-halpe26_700e-256x192-4d3e73dd_20230605.zip',
         'pose_input_size':[192,256]}"""
  
  # With one-stage RTMO model 
  # Requires pose_model = 'Body'. Marker augmentation won't work, Kinematic analysis will
  mode = """{'pose_class':'RTMO', 
         'pose_model':'https://download.openmmlab.com/mmpose/v1/projects/rtmo/onnx_sdk/rtmo-m_16xb16-600e_body7-640x640-39e78cc4_20231211.zip', 
         'pose_input_size':[640, 640]}"""
  
  # With animal pose estimation:
  # Marker augmentation won't work, and you will need to create your own OpenSim skeleton for kinematic analysis.
  mode = """{'pose_class':'RTMPose',
         'pose_model':'https://download.openmmlab.com/mmpose/v1/projects/rtmposev1/onnx_sdk/rtmpose-m_simcc-ap10k_pt-aic-coco_210e-256x256-7a041aa1_20230206.zip',
         'pose_input_size':[256,256]}"""
  
  # Same approach for hand or face pose estimation, check the RTMLib documentation for more information.
  

[divan]

Thanks @davidpagnon. I'm trying to make sense of the models/tools landscape myself and so far it's hard to tell whether it's my lack of knowledge/terms, or if it's just messy :) Speaking of monocular videos, I found this example from EasyMocap, which seems to be doing exactly what I need (in fact, my main sport is figure skating):

I tried to use this repo – it was absolute mess, requiring to download manually files from dropbox/onedrive/other filesharing platforms and somehow cook together, and some files aren't available anymore. Not usable at all, but at least it gives hope that the open-source models are already good enough.

I also tried Meta's Sapiens models and they're worked well, but I'm struggling to understand how to use their output into anything I can use in Blender...

[davidpagnon]

Hi @divan, is there any chance you could join the Discord server so that we can keep talking on a more appropriate medium? If not, don't hesitate to open a dedicated issue (even though there is no real "issue")