GigaTIME: Multimodal AI generates virtual population for tumor microenvironment modeling (Cell)

Official implementation of GigaTIME

📄 Paper • 🤗 Model Card

Environment Setup

We recommend using Conda for environment management. The codebase has been tested with Python 3.11 using A100 GPUs for optimal reproducibility. Before creating the environment, ensure that the torch version specified in environment.yml matches your GPU and CUDA driver setup.

To set up the environment, run:

conda env create -f environment.yml

This will create a Conda environment named gigatime. Activate it with:

conda activate gigatime

Data

A set of 50 paired H&E and mIF patches from the test set is available for evaluation. Download the sample data from Dropbox.

After downloading, unzip the folder and place it in the data directory:

unzip sample_test_data.zip -d ./data/

Make sure the extracted folder are located in ./data/.

Pre-trained Model

Model card available in HuggingFace

You need to agree to the terms to access the models. Once you have the necessary access, set your HuggingFace read-only token as an environment variable:

export HF_TOKEN=<huggingface read-only token>

If you don’t set the token, you might encounter the following error:

ValueError: We have no connection or you passed local_files_only, so force_download is not an accepted option.

Once that is done, you can load your model like this:

from huggingface_hub import snapshot_download
import torch

repo_id = "prov-gigatime/GigaTIME"
local_dir = snapshot_download(repo_id=repo_id)

weights_path = os.path.join(local_dir, "model.pth")
state_dict = torch.load(weights_path, map_location="cpu")
model.load_state_dict(state_dict)

Tutorials

• Inference Tutorial:

Learn how to load the model and run predictions on sample patches: scripts/gigatime_testing.ipynb

• Training Tutorial:

Understand the training workflow with a one-epoch demo: scripts/gigatime_training.ipynb

Training GigaTIME cross-modal translator

We also release the script needed to train the GigaTIME model here.

To train the model:

python scripts/db_train.py --arch gigatime   --tiling_dir "gigatime_training_path"  --window_size 256       --batch_size 32     --sampling_prob 1     --name GigaTIME_model    --output_dir "Output_Directory"    --epoch 300 --input_h 512 --input_w 512 --lr 0.001 --loss BCEDiceLoss --val_sampling_prob 1 --num_workers 12 --gpu_ids 0 1 2 3 4 5 6 7 --crop True --metadata "Gigatime metadata file"

Model Uses

Intended Use

The data, code, and model checkpoints are intended to be used solely for (I) future research on pathology AI models and (II) reproducibility of the experimental results reported in the reference paper. The data, code, and model checkpoints are not intended to be used in clinical care or for any clinical decision-making purposes.

Primary Intended Use

The primary intended use is to support AI researchers reproducing and building on top of this work. GigaTIME should be helpful for generating virtual mIF profiles from routine H&E pathology slides.

Out-of-Scope Use

Any deployed use case of the model --- commercial or otherwise --- is out of scope. Although we evaluated the models using a broad set of publicly-available research benchmarks, the models and evaluations are intended for research use only and not intended for deployed use cases.

License Notice

The model is not intended or made available for clinical use as a medical device, clinical support, diagnostic tool, or other technology intended to be used in the diagnosis, cure, mitigation, treatment, or prevention of disease or other conditions. The model is not designed or intended to be a substitute for professional medical advice, diagnosis, treatment, or judgment and should not be used as such. All users are responsible for reviewing the output of the developed model to determine whether the model meets the user’s needs and for validating and evaluating the model before any clinical use.

Citation

@article{valanarasu2025multimodal,
  title={Multimodal AI generates virtual population for tumor microenvironment modeling},
  author={Valanarasu, Jeya Maria Jose and Xu, Hanwen and Usuyama, Naoto and Kim, Chanwoo and Wong, Cliff and Argaw, Peniel and Shimol, Racheli Ben and Crabtree, Angela and Matlock, Kevin and Bartlett, Alexandra Q and others},
  journal={Cell},
  year={2025},
  publisher={Elsevier}
}