TRex pose estimation does not match yolo skeleton

[FedericoCM]

Describe the bug
Hi Tristan,

I am trying to track flies with pose estimation using a yolo11 model that I trained on my dataset. My fly skeleton has just 2 points (0:head, 1:tail). In the TRex GUI, I can see in "raw" mode that the keypoints from the model identify head and tail correctly. On the other hand, in "tracking" mode I noticed that the midline calculated by TRex doesn't follow the skeleton points of yolo and head and tail flip quite frequently back and forth. This flickering is evident when I plot the positional data of the head compared with wcentroid and pcentroid.

I tried to solve the problem using "pose_midline_indexes" = [0, 1]" and "detect_skeleton" = ["fly",[[0,1]]] but it doesn't seem to have an effect.

In the output .csv files, I noticed that TRex export 16 keypoints, that is the default "human" skeleton. The variables X0, X1, Y0, Y1 correctly show the coordinates of my 2 yolo points, but X2-16, Y2-16 show "nan". I looked at the posture data and the midline shows 25 points according to default. I don't know if this can be useful.

To Reproduce

1. Created a new environment with conda-forge conda create -n trexia -c trex-beta trex
2. conda activate trexia
3. Executed trex with video and settings file, yolo model path specified in settings. Output redirected to trex.log. All files are attached
   trex -i C:\Users\fec45kv\Videos\video_trial.avi -s C:\Users\fec45kv\Videos\trex2.settings > C:\Users\fec45kv\Videos\trex.log

Expected behavior (if it is not a simple crash)
I would expect TRex to follow the skeleton of my yolo model

Screenshots
Entire screen

Images that show the problem

raw1

raw2

trex1

trex2

Positional data

Computer (please complete the following information):
Edition Windows 10 Enterprise
Version 22H2
Installed on ‎28/‎03/‎2022
OS build 19045.5371
Experience Windows Feature Experience Pack 1000.19060.1000.0

Processor Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz 3.00 GHz
Installed RAM 32.0 GB (31.8 GB usable)
System type 64-bit operating system, x64-based processor
Graphic card NVIDIA Quadro P620
Terminal used Miniforge Prompt

Additional context
Video
video_trial.zip
Log
trex.log
Settings files and yolo model
settings_model.zip
PV file and results
pv_results.zip
Output data
output.zip

[mooch443]

Thank you so much for adding those two issues! I'll have a look later or tomorrow :)

[mooch443]

Alright, so the pose (this) issue seems to be resolved fairly easily. Will be fixed in the next patch -- that is if you also change outline_resample = 0.5, although I don't think it's strictly necessary. I would do it anyway because your animals are fairly small and thus we only get a few outline points, which are then smoothed and so on - we just need a few more points to work with. This could also help a bit with your version already, although not much. The mistake on my part was a typical "off by one" error. Plus some fun other stuff I found!

The midline / outline generated by TRex is conceptually different from the raw pose data. N=25 number of midline points seems to be the default, which you can change of course (midline_resolution). I wouldn't use it if you want to rely purely on pose (keypoints by YOLO). You could just use your poseX and poseY. I will change this so it will only export keypoints that are reported to exist by YOLO. This is probably a good idea. I will need an option to disable this, so I guess it will be called "output_auto_pose".

[FedericoCM]

Thank you very much for your effort. I tried with outline_resample = 0.5 and it seems to work very well in my trial video. I would still wait for the new version to test TRex on my dataset but it seems already a huge improvement.