Using MedSAM for recon of vertebrae

[nachoXXL]

Hi all,
We're trying to fine-tune SAM With MedSAM to find vetrebrae o f dogs. However, it does not seem to learn anything.
We did some changes to the code but the training loop is kept as is.
There is one point where we've our doubts and maybe someone can clarify the issue:

In train the loss (seg_loss) is calculated on a non-binary mask (before mask_predictions > 0.5) however
in validation the loss is calculated after 'mask_predictions > 0.5'. This seems inconsistent.
However when we try to change this (calc loss after >) we loose the computational graph whatever we try.

So the questions we have:

• is the loss calculation with a non-binary mask detection correct?
• if so: why is the validation different?

Kind regards,
Nacho

[JunMa11]

Hi,

Both SAM/MedSAM are essentially point/bounding box-based segmentation methods. For your task, nnunet would be a better choice.
https://github.com/MIC-DKFZ/nnUNet

For your questions

1. Yes, during training, we don't threshold the probability.
2. During training, we need to back-propagate the loss. so we don't need to convert the probability map to a binary mask. However, during validation/inference, we need this thresholding operation to get a binary mask as the final segmentation results.