Text-Guided Brain Tumor Segmentation

Using Vision-Language Models with BraTS 2020 FLAIR MRI and Radiology Reports

Project: Multimodal Brain Tumor Segmentation | BraTS 2020 + TextBraTS
Architecture: CLIP-Inspired Multimodal UNet (ResNet34 + DistilBERT)
Framework: PyTorch

Overview

This project implements a scientifically rigorously validated multimodal pipeline that fuses textual radiology priors (reports) with visual MRI features (FLAIR slices) to improve brain tumor segmentation.

Key Features

• Zero Data Leakage: Patient-level dataset splitting (80/20).
• Corrected Thresholding: Metrics are computed using rigorous binary thresholding ($\tau = 0.5$).
• Architecture: Image features are extracted using ResNet34 and Text features using DistilBERT. The representations are merged into a Unet bottleneck.
• Explainability: True Grad-CAM implementation hooked into the ResNet bottleneck to visualize spatial attention shifts due to text prior.

Files

• DL_Final_project_notebook.ipynb: The main research notebook.

Requirements

Install the required packages using:

pip install -r requirements.txt

Dataset

This project expects the TextBraTSData and FLAIR_BRATS2020_split datasets. Due to file size limits, these datasets are not hosted in this repository. Place them in the appropriate directory or update the dataset paths in the notebook before running.