Merlin: Vision Language Foundation Model for 3D Computed Tomography

                

Merlin is a 3D VLM for computed tomography that leverages both structured electronic health records (EHR) and unstructured radiology reports for pretraining.

⚡️ Installation

To install Merlin, you can simply run:

pip install merlin-vlm

For an editable installation, use the following commands to clone and install this repository.

conda create --name merlin python==3.10
conda activate merlin

git clone https://github.com/StanfordMIMI/Merlin.git
cd Merlin
pip install -e .

# Alternatively, to install exact package versions as tested:
# uv sync

🚀 Inference with Merlin

To create a Merlin model with both image and text embeddings enabled, use the following:

from merlin import Merlin

model = Merlin()

To initialize the model with only image embeddings active, use:

from merlin import Merlin

model = Merlin(ImageEmbedding=True)

To initialize the model for phenotype classification, use:

from merlin import Merlin

model = Merlin(PhenotypeCls=True)

To initialize the model for five-year disease prediction, use:

from merlin import Merlin

model = Merlin(FiveYearPred=True)

To initialize the model for radiology report generation, use:

from merlin import Merlin

model = Merlin(RadiologyReport=True)

For inference on a demo CT scan, please check out the general demo and report generation demo.

For additional information, please read the inference documentation and report generation documentation.

For segmentation, we integrated Merlin with nnU-Net framework. Please refer to the Merlin segmentation repository and its README for detailed setup and inference instructions.

📂 Merlin Abdominal CT Dataset

We are excited to release the Merlin Abdominal CT Dataset to the community!

For details on accessing and using the dataset, please see the download documentation!

📎 Citation

If you find this repository useful for your work, please cite the cite the original paper:

@article{blankemeier2024merlin,
  title={Merlin: A vision language foundation model for 3d computed tomography},
  author={Blankemeier, Louis and Cohen, Joseph Paul and Kumar, Ashwin and Van Veen, Dave and Gardezi, Syed Jamal Safdar and Paschali, Magdalini and Chen, Zhihong and Delbrouck, Jean-Benoit and Reis, Eduardo and Truyts, Cesar and others},
  journal={Research Square},
  pages={rs--3},
  year={2024}
}