Master's Thesis: Assessing Fairness in Mammography-Based Breast Cancer Detection Models

This repository contains the code that I used to conduct the experiments of my thesis.

Prerequisites

Data Access

You need access to the optimam-data-eu4 Google Cloud Storage bucket, which contains images and metadata from the OPTIMAM Database (OMI-DB):

optimam-data-eu4/
├── IMAGES/              # Raw DICOM images
├── DATA/                # Additional metadata
├── omidb.csv            # OMI-DB as `.csv`.
├── clients_info.parquet # Contains `Ethnicity`, `Age`, and `IMD` metadata
├── ffdm.parquet         # Contains `Manufacturer` metadata

Contact your supervisor for bucket access permissions.

Compute Environment

Training and inference (testing) requires GPU resources. We use Google Cloud VMs with persistent storage.

Setup instructions: See docs/google-cloud-setup-instruction.md for:

• Creating persistent disks
• Configuring VM templates (CPU/GPU)
• Setting up GitHub deploy keys
• Post-launch scripts

Local Development

For local development (no training), you need:

• Python 3.11+
• uv package manager

# macOS
uv sync --extra macos

# Linux (CPU)
uv sync --extra linux-cpu

Important: All scripts must be run from the project root directory. The project structure uses relative imports that expect the root as the working directory.

Data Preparation

This section describes how to prepare the dataset from raw OMI-DB data.

Overview

The data preparation pipeline transforms raw DICOM images into preprocessed .npz files with associated metadata:

omidb.csv (raw)
    ↓  filter & create index
index_filtered.csv
    ↓  preprocess images
index_preprocessed.csv + .npz files
    ↓  merge metadata
index_with_metadata.csv
    ↓  add exposure (proxy for breast density) & derived columns
index_final.csv

Step 1: Filter and Create Index

Filter OMI-DB to include only:

• PresentationIntentType = "FOR PRESENTATION"
• ViewPosition in ["CC", "MLO"]
• EpisodeStatus in ["N", "M", "B", "CI"]

Note: Episodes with status "NAB" (normal client information) were excluded due to incomplete OMI-DB documentation. These cases (~0.3% of the dataset) are equivalent to "Normal" and should have been included.

python scripts/data_preparation/create_index.py \
    --omidb_csv /path/to/omidb.csv \
    --output_csv data/index_filtered.csv

Output: data/index_filtered.csv with columns image_path, label, client_id, episode_id, classification, study_instance_uid, sop_instance_uid

Step 2: Preprocess Images

Apply preprocessing pipeline (DICOM read → flip → crop → pad → resize → normalize):

python scripts/data_preparation/preprocess_images.py \
    --images_path /path/to/IMAGES \
    --csv_path data/index_filtered.csv \
    --output_path data/preprocessed

Output:

• Preprocessed images as .npz files in data/preprocessed/ directory (shape: 1×1024×768, dtype: float16)
• data/preprocessed/index_preprocessed.csv with added preprocessed_image_path column and crop coordinates

Note: This step may take a while. To speed it up, the script leaves one cpu core for the system and all other available cpu cores to process the images in parallel. The more CPU cores for the machine, the faster.

Step 3: Merge Metadata

Add demographic attributes from parquet files:

python scripts/data_preparation/merge_metadata.py \
    --dataset_csv data/preprocessed/index_preprocessed.csv \
    --ffdm_parquet /path/to/ffdm.parquet \
    --clients_info_parquet /path/to/clients_info.parquet \
    --output_csv data/index_with_metadata.csv

Output: data/index_with_metadata.csv with added columns Manufacturer, Model, AgeAtScreening, EthnicCategory, IMD_decile

Step 4: Add Exposure (Breast Density Proxy)

Extract exposure values from DICOM headers:

python scripts/data_preparation/add_exposure_column.py \
    --csv data/index_with_metadata.csv \
    --image_folder /path/to/IMAGES \
    --output data/index_with_exposure.csv

Step 5: Add Derived Columns

Add binned age groups and exposure deciles for fairness analysis:

python scripts/data_preparation/add_derived_columns.py \
    --input_csv data/index_with_exposure.csv \
    --output_csv data/index_final.csv

Output: data/index_final.csv with added Age_Group and Exposure_Decile columns

This is your complete dataset ready for training!

Training

Train binary classification models (Normal vs Abnormal) for single view mammography using PyTorch Lightning with Weights & Biases for experiment tracking.

Basic Usage

python scripts/train.py \
    --backbone shufflenet \
    --head small \
    --csv_path data/index_final.csv \
    --images_root data/preprocessed \
    --experiment_name my_experiment \
    --learning_rate 1.5e-5 \
    --batch_size 256 \
    --max_epochs 50

Model Architectures

We train three lightweight architectures selected for their successful application to mammography tasks and available pretrained weights:

--backbone	Architecture	Feature dim
shufflenet (default)	ShuffleNet V2 x1.0	1024
resnet34	ResNet-34	512
mobilenet_v3	MobileNetV3 Large	960

--head	Description
small (default)	43→43→1 (from Ellis et al.)
large	feature_dim→feature_dim→1

All models use ImageNet-pretrained weights adapted for single-channel (grayscale) input by averaging the RGB channel weights.

Training Hyperparameters

Learning rates and batch sizes were tuned per architecture through manual exploration. The table shows the hyperparameters used in the thesis experiments:

Parameter	ShuffleNet	ResNet-34	MobileNetV3
Learning rate	1.5e-5	1e-6	1e-5
Batch size	256	64	64
Optimizer	AdamW	AdamW	AdamW
Weight decay	0.01	0.01	0.01
Scheduler	OneCycle	OneCycle	OneCycle
Max epochs	50	50	50
Early stopping patience	20	20	20

Training Behavior

• Loss function: Binary cross-entropy with class weights (pos_weight ≈ 2.7) to handle label imbalance
• Early stopping: Training stops if validation AUPRC doesn't improve for 20 epochs
• Checkpointing: Top 3 models saved based on validation AUPRC to checkpoints/{experiment_name}/
• Experiment tracking: Logs to Weights & Biases (disable with --no_wandb)

Fairness-Augmented Training (ShuffleNet only)

Standard training uses class weights to handle label imbalance. Fairness-augmented training additionally applies per-sample weights based on demographic groups (age) to encourage equitable performance across subgroups.

Step 1: Generate age bucket weights

The age weights file maps age groups to sample weights using inverse frequency weighting. Generate it from your training data:

python scripts/utilities/compute_fairness_weights.py \
    --csv_path data/index_final.csv \
    --age_col AgeAtScreening \
    --train_split 0.6 \
    --val_split 0.2 \
    --seed 42 \
    --output data/age_bucket_weights.json

This computes weights as: weight = (total_samples / num_buckets) / bucket_count for each age group. The script uses the same client-based split as training (seed 42) to calculate weights only from the training set.

Step 2: Train with fairness weighting

python scripts/train.py \
    --backbone shufflenet \
    --csv_path data/index_final.csv \
    --images_root data/preprocessed \
    --age_weights_path data/age_bucket_weights.json \
    --learning_rate 1.5e-5 \
    --batch_size 256 \
    --experiment_name shufflenet_fair

During training, each sample's loss is scaled by: age_group_weight * class_weight.

Evaluation

Evaluate a trained model checkpoint on the test set only and generate predictions with metadata for fairness analysis.

Finding Your Best Checkpoint

After training, checkpoints are saved to checkpoints/{experiment_name}/. The ModelCheckpoint callback saves:

• Top 3 models by validation AUPRC: epoch=XX-val_auprc=Y.YYY.ckpt
• Last checkpoint: last.ckpt

Check the training output or W&B logs to identify the best checkpoint path.

Running Evaluation

python scripts/test.py \
    --backbone shufflenet \
    --head small \
    --checkpoint checkpoints/my_experiment/epoch=15-val_auprc=0.523.ckpt \
    --csv_path data/index_final.csv \
    --images_root data/preprocessed \
    --output_csv results/test_results/my_model/predictions.csv \
    --merge_source_csv data/index_final.csv \
    --model_id my_model \
    --experiment_name my_experiment_test

Key Parameters

• --backbone and --head: Must exactly match the architecture used during training
• --checkpoint: Path to the checkpoint file
• --csv_path: CSV used to determine train/val/test split (should match training, default: data/index_final.csv)
• --merge_source_csv: Full metadata CSV to merge with predictions (typically same as --csv_path)
• --model_id: Identifier used to name the merged output CSV (defaults to --experiment_name, or "model" if neither provided)
• --experiment_name: W&B run name (optional, only required if W&B logging is enabled)
• --no_wandb: Disable W&B logging (useful for local testing)

What This Does

1. Loads model from checkpoint (using specified backbone and head architecture)
2. Determines test split using client-based splitting (default seed: 42, same as training)
3. Generates predictions on test set only
4. Saves raw predictions to --output_csv with columns: image_path, label, pred, prob
5. Merges predictions with metadata from --merge_source_csv by matching on preprocessed_image_path (if --merge_source_csv is provided)
6. Saves merged CSV in the same directory as --output_csv, named {original_stem}_{model_id}.csv
   • Example: With --output_csv results/test_results/my_model/predictions.csv and --model_id my_model, the merged file is results/test_results/my_model/index_final_my_model.csv
7. Logs to W&B for experiment tracking (disable with --no_wandb)

Output Files

Using the example above:

• results/test_results/my_model/predictions.csv - Raw test set predictions only
• results/test_results/my_model/index_final_my_model.csv - Test set with predictions and all metadata columns (used for fairness analysis)

Important: The merged CSV contains only test set samples (not train/val). Fairness analysis is performed exclusively on the test set to ensure unbiased evaluation. This CSV includes predictions alongside protected attributes (age, ethnicity, IMD, manufacturer, breast density).

Fairness Analysis

After generating predictions with test.py, analyze model fairness across protected attributes. The fairness analysis evaluates whether model performance varies systematically across age, ethnicity, socioeconomic status, breast density, and manufacturer.

Creating a Model Configuration File

Create a JSON file listing models to evaluate. Each entry needs the path to merged predictions CSV (from test.py) and a model identifier.

Single model example (my_model_config.json):

[
  {
    "model_results_path": "results/test_results/my_model/index_final_my_model.csv",
    "model_id": "my_model"
  }
]

Multi-model comparison example (all_models_config.json):

[
  {
    "model_results_path": "results/test_results/shufflenet/index_final_shufflenet.csv",
    "model_id": "shufflenet"
  },
  {
    "model_results_path": "results/test_results/resnet34/index_final_resnet34.csv",
    "model_id": "resnet34"
  },
  {
    "model_results_path": "results/test_results/mobilenet_v3/index_final_mobilenet_v3.csv",
    "model_id": "mobilenet_v3"
  }
]

The model_results_path should point to the merged CSV created by test.py (see "Output Files" in the Evaluation section above).

Running Fairness Analysis

python scripts/fairness_analysis/compare_models.py \
    --model-configs my_model_config.json \
    --results-dir results/fairness_analysis \
    --output-dir results/model_comparison

What This Analyzes

The fairness analysis pipeline:

1. Loads test set predictions with protected attributes from merged CSV

2. Normalizes metadata into standardized bins

3. Creates slices for each protected attribute:

   • Age: 0-49, 50-54, 55-59, 60-64, 65-69, 70+
   • Ethnicity: White, Unknown, Asian/Asian British, Black/Black British/Caribbean/African, Not stated, Any other ethnic group, Mixed/Multiple ethnic groups
   • IMD Decile: 1 (most deprived) to 10 (least deprived), Unknown
   • Breast Density: Low Density, Low-Moderate Density, Moderate-High Density, High Density (exposure quartiles)
   • Manufacturer: HOLOGIC Inc., SIEMENS, GE MEDICAL SYSTEMS

4. Computes performance metrics per slice:

   • AUPRC (primary metric)
   • Recall (TPR)
   • FPR
   • TPR at fixed FPR (0.15, 0.3)
   • FPR at fixed TPR (0.6, 0.8)

5. Computes fairness metrics per protected attribute:

   • Min: Lowest metric value across slices
   • Max: Highest metric value across slices
   • Range: Max - Min (disparity between extremes)
   • Std Deviation: Variability across slices
   • Difference of Equalized Odds: Average of TPR difference and FPR difference

6. Saves structured results as CSV files in results/fairness_analysis/

7. Generates visualizations in results/model_comparison/:

   • Overall performance table comparing all models
   • Subgroup performance plots (one per metric per attribute)
   • Fairness metrics comparison plots (min/max/range/std per metric)

Output Files

Results directory (results/fairness_analysis/):

• {model_id}_overall_performance.csv - Overall test set metrics
• {model_id}_slice_results_{attribute}.csv - Per-slice metrics for each attribute
• {model_id}_fairness_metrics_{attribute}.csv - Aggregated fairness metrics per attribute

Visualizations directory (results/model_comparison/):

• overall_performance_table.png - Side-by-side model comparison table
• subgroup_{attribute}_{metric}.png - Performance across slices (e.g., subgroup_AgeAtScreening_AUPRC.png)
• fairness_{aggregation}_{metric}.png - Fairness metrics comparison (e.g., fairness_range_AUPRC.png)

Understanding Correlations

Analyze correlations between protected attributes and outcomes:

Chi-square tests (categorical vs categorical):

python scripts/fairness_analysis/correlation_tests/run_chi_square_tests.py \
    --input_csv data/index_final.csv

Kruskal-Wallis tests (categorical vs numerical):

python scripts/fairness_analysis/correlation_tests/run_kruskal_wallis_tests.py \
    --input_csv data/index_final.csv

Spearman correlations (numerical vs numerical):

python scripts/fairness_analysis/correlation_tests/run_spearman_correlation_tests.py \
    --input_csv data/index_final.csv

Results are saved to results/correlation_tests/ with corresponding visualizations automatically generated.

Performance Metric Tables

Generate detailed performance metric tables for individual protected attributes:

python scripts/visualization/slice_analysis/plot_performance_metric_table.py \
    --model_results_csv results/test_results/my_model/index_final_my_model.csv \
    --model_id my_model \
    --metadata_column AgeAtScreening \
    --output_path results/tables/age_performance_table.png

Parameters:

• --model_results_csv: Path to merged CSV with predictions and metadata
• --model_id: Model identifier for the title
• --metadata_column: Column to analyze (AgeAtScreening, EthnicCategory, Manufacturer, IMD_decile)
• --output_path: Where to save the table image (optional, displays if not provided)

Output: Publication-quality table showing AUPRC, Recall, FPR, and operating points for each slice.

Re-generating Visualizations

To regenerate visualizations without re-running evaluation:

python scripts/fairness_analysis/compare_models.py \
    --results-dir results/fairness_analysis \
    --output-dir results/model_comparison \
    --skip-evaluation

End-to-End Example

Here's a complete workflow from data preparation through fairness analysis for a single model:

# 1. DATA PREPARATION (run once)
# Assuming you have access to optimam-data-eu4 bucket with omidb.csv and parquet files

# Step 1: Create filtered index
python scripts/data_preparation/create_index.py \
    --omidb_csv /path/to/omidb.csv \
    --output_csv data/index_filtered.csv

# Step 2: Preprocess images (this takes a while)
python scripts/data_preparation/preprocess_images.py \
    --images_path /path/to/IMAGES \
    --csv_path data/index_filtered.csv \
    --output_path data/preprocessed

# Step 3: Merge metadata
python scripts/data_preparation/merge_metadata.py \
    --dataset_csv data/preprocessed/index_preprocessed.csv \
    --ffdm_parquet /path/to/ffdm.parquet \
    --clients_info_parquet /path/to/clients_info.parquet \
    --output_csv data/index_with_metadata.csv

# Step 4: Add exposure (breast density proxy)
python scripts/data_preparation/add_exposure_column.py \
    --csv data/index_with_metadata.csv \
    --image_folder /path/to/IMAGES \
    --output data/index_with_exposure.csv

# Step 5: Add derived columns for fairness analysis
python scripts/data_preparation/add_derived_columns.py \
    --input_csv data/index_with_exposure.csv \
    --output_csv data/index_final.csv

# 2. TRAINING
python scripts/train.py \
    --backbone shufflenet \
    --head small \
    --csv_path data/index_final.csv \
    --images_root data/preprocessed \
    --learning_rate 1.5e-5 \
    --batch_size 256 \
    --max_epochs 50 \
    --experiment_name my_shufflenet_experiment

# Training will create checkpoints in: checkpoints/my_shufflenet_experiment/
# Note the best checkpoint filename from training output or W&B

# 3. EVALUATION
# Use the best checkpoint (e.g., epoch=15-val_auprc=0.523.ckpt)
mkdir -p results/test_results/my_shufflenet

python scripts/test.py \
    --backbone shufflenet \
    --head small \
    --checkpoint checkpoints/my_shufflenet_experiment/epoch=15-val_auprc=0.523.ckpt \
    --csv_path data/index_final.csv \
    --images_root data/preprocessed \
    --output_csv results/test_results/my_shufflenet/predictions.csv \
    --merge_source_csv data/index_final.csv \
    --model_id my_shufflenet \
    --experiment_name my_shufflenet_test

# This creates:
# - results/test_results/my_shufflenet/predictions.csv (raw predictions)
# - results/test_results/my_shufflenet/index_final_my_shufflenet.csv (predictions merged with metadata - used for fairness)

# 4. FAIRNESS ANALYSIS
# Create a config file
cat > my_model_config.json << EOF
[
  {
    "model_results_path": "results/test_results/my_shufflenet/index_final_my_shufflenet.csv",
    "model_id": "my_shufflenet"
  }
]
EOF

# Run fairness analysis
python scripts/fairness_analysis/compare_models.py \
    --model-configs my_model_config.json \
    --results-dir results/fairness_analysis \
    --output-dir results/model_comparison

# Results are in:
# - results/fairness_analysis/ (CSV files with metrics)
# - results/model_comparison/ (visualization plots)

# 5. OPTIONAL: Generate detailed performance tables
python scripts/visualization/slice_analysis/plot_performance_metric_table.py \
    --model_results_csv results/test_results/my_shufflenet/index_final_my_shufflenet.csv \
    --model_id my_shufflenet \
    --metadata_column AgeAtScreening \
    --output_path results/tables/my_shufflenet_age_performance.png

Training Multiple Models for Comparison

To compare different architectures:

# Train ResNet-34
python scripts/train.py \
    --backbone resnet34 \
    --head small \
    --csv_path data/index_final.csv \
    --images_root data/preprocessed \
    --learning_rate 1e-6 \
    --batch_size 64 \
    --max_epochs 50 \
    --experiment_name my_resnet34_experiment

# Train MobileNet V3
python scripts/train.py \
    --backbone mobilenet_v3 \
    --head small \
    --csv_path data/index_final.csv \
    --images_root data/preprocessed \
    --learning_rate 1e-5 \
    --batch_size 64 \
    --max_epochs 50 \
    --experiment_name my_mobilenet_experiment

# Evaluate each model (repeat test.py with appropriate backbone/checkpoint)
# Then create a multi-model config:
cat > all_models_config.json << EOF
[
  {
    "model_results_path": "results/test_results/my_shufflenet/index_final_my_shufflenet.csv",
    "model_id": "my_shufflenet"
  },
  {
    "model_results_path": "results/test_results/my_resnet34/index_final_my_resnet34.csv",
    "model_id": "my_resnet34"
  },
  {
    "model_results_path": "results/test_results/my_mobilenet/index_final_my_mobilenet.csv",
    "model_id": "my_mobilenet"
  }
]
EOF

# Run comparison analysis
python scripts/fairness_analysis/compare_models.py \
    --model-configs all_models_config.json \
    --results-dir results/fairness_analysis \
    --output-dir results/model_comparison