How is the input segmentation updated during tracking and can that be tuned?

[ATB33]

Hi,

I have a general question regarding how Ultrack processes the data to track cells. The way I understand it is that you provide a foreground area which can be cells, and contours wich correspond to the edges of the segments. During the tracking the contours might be changed.

Now we segment our images with cellpose before doing the tracking, and the segmentation is pretty accurate. We tried to use the track() function and insert the segmentation as labels, but we always get an error:

TypeError: Implicit conversion to a NumPy array is not allowed. Please use .get() to construct a NumPy array explicitly.

So instead, we used the segmentation as foreground and used skimage.segmentation.find_boundaries(foreground[z].get(), mode="outer") to obtain the edges, and then we run track(config, foreground = segmentation, edges = contours_array, overwrite = True). However, during the tracking there are always cell segments that are fused together, sometimes more than 10 segments to a single one. Our samples have very dense cells, so a lot of the cells touch and therefore have the same edge.

There are two things I am unsure about:

• In the documentation, it says the foreground is a segmentation probability. Does that mean all pixel values should be between 0 and 1? If so, then I would wonder why it works the way we do it.
• Is there a way to avoid Ultrack changing the segmentation during tracking? I understand that this is one of the strenghts that you can use the tracking to improve the segmentation, but since our segmentation is quite good already I would like to tone that down. I tried that using parameters such as the max_area (as described in optimizing), but then the fused cells would just disappear. Do you have any tipps on that? How is the input segmentation influenced by the tracking?

Thanks in advance for your time!

[JoOkuma]

Hey @ATB33,

I have a general question regarding how Ultrack processes the data to track cells. The way I understand it is that you provide a foreground area which can be cells, and contours which correspond to the edges of the segments.

Correct

During the tracking the contours might be changed.

Could you clarify what you mean here?

Now we segment our images with cellpose before doing the tracking, and the segmentation is pretty accurate. We tried to use the track() function and insert the segmentation as labels, but we always get an error:
TypeError: Implicit conversion to a NumPy array is not allowed. Please use .get() to construct a NumPy array explicitly.

Could you try installing cucim? I think our heuristic to guess if we use a cuda acceleration or not is failing.

So instead, we used the segmentation as foreground and used skimage.segmentation.find_boundaries(foreground[z].get(), mode="outer") to obtain the edges, and then we run track(config, foreground = segmentation, edges = contours_array, overwrite = True).

This is very similar to what we do and should be correct.

Does that mean all pixel values should be between 0 and 1? If so, then I would wonder why it works the way we do it.

This is not a requirement, while we mention probability to provide some intuition, a notion of ranking is the most appropriate.

Is there a way to avoid Ultrack changing the segmentation during tracking? I understand that this is one of the strenghts that you can use the tracking to improve the segmentation, but since our segmentation is quite good already I would like to tone that down. I tried that using parameters such as the max_area (as described in optimizing), but then the fused cells would just disappear.

The only way to avoid the segmentation hypotheses is to manually add your nodes with add_new_nodes, you can get the mask and other content from skimage's regionprops.

Still, I'm surprised it disappears when you decrease the maximum area.
I would try reducing the minimum area to make sure you're not accidentally removing the real segments.

Also, check if your linking_config.max_distance is large enough between the centroids of the cells.

Are you missing a lot of segments? Could you try to set the tracking_config.<disappear|appear>_weight to zero or very close to zero as -0.00001 to check if they are being removed because are too much penalized.