Skip to content

README.md

Latest commit

 

History

History
403 lines (327 loc) · 18.5 KB

README.md

File metadata and controls

403 lines (327 loc) · 18.5 KB
Raw

mlops

Open-source MLOps framework for continuous monitoring and lifecycle management of AI models in medical imaging. Supports drift detection, embedding-based monitoring, MLflow integration, automated validation, and reproducible deployment pipelines for reliable clinical AI systems.

Installation of the environment wsl on Windows

Wsl installation

Install wsl (with the Ubuntu distribution) on your Windows dev machine or use Ubuntu instead. Make sure that you do a restart after you installed wsl.

ssh-key setup

To generate an ssh-key, use ssh-keygen -t ecdsa in the .ssh folder of your Windows user account, e.g., C:\Users\{USER}\.ssh. Then open the Windows explorer and type \wsl.localhost\Ubuntu\home{USER} in the explorer. If there is not already a folder .ssh, create one. Then copy the generated ssh keys from C:\Users{USER}.ssh to the .ssh folder in the wsl file system. Please add your public ssh key to your GitHub account.

Installation of Pycharm

Install pycharm and run it as admin. As you have previously installed wsl and restarted the computer, there should be an Ubuntu terminal based on wsl available in your Pycharm. If so, please open this terminal. If not, please check your wsl installation. You can check with pwd if the selected directory in your Ubuntu terminal session is /home/{USER}. If not, navigate there and perform the following steps.

Installation of mlops virtual environment (for development)

  1. Clone the repository https://github.com/Akros-STM/mlops via git in the wsl filesystem, e.g., under /home/{USER}.
  2. Checkout the branch develop.
  3. Delete the folder .vitualenvs in /home/{USER} if it exists.
  4. Navigate to the root directory of your project /home/{USER}/mlops.
  5. Create a virtual environment via python3.12 -m venv .venv. If step 4 does not work, you may need to install the venv extension for your python, i.e., by sudo apt install python3.X-venv. After the installation, try again to create a virtual environment.
  6. Using wsl, activate the virtual environment by source .venv/bin/activate.
  7. Update pip via pip install --upgrade pip. Also, install wheel via pip install wheel and the setuptools via pip install setuptools.
  8. Install poetry in the virtual environment via pip install poetry.
  9. Install the dependencies via poetry install --no-root --with dev.
  10. If you have no GPU available, then install pytorch via poetry run pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu. If you have a GPU available, then execute the command poetry run pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121.
  11. Go to the Pycharm Settings, then the Project: mlops, and then the Project Interpreter.
  12. Click on Add interpreter and select On WSL.
  13. Add an existing python interpreter and select the python in the previously created venv /home/{USER}/mlops/venv/bin/python3.X.
  14. Run the Python scripts from the root directory in the terminal or use the Pycharm Run/Debug functionality. Remember to set the Working directory in the Run/ Debug configuration to the project root directory.

Installation of mlops as a docker container

Installation of docker on Ubuntu WLS

  1. Go in the WSL Ubuntu terminal and execute sudo apt update and sudo apt install -y ca-certificates curl gnupg.
  2. Create a keyrings directory with certain permissions via sudo install -m 0755 -d /etc/apt/keyrings.
  3. Download the Docker GPG key for Linux and store it in a format supported by apt with curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \ sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
  4. Set the permissions of the docker gpg file via sudo chmod a+r /etc/apt/keyrings/docker.gpg.
  5. Add the official docker packet repository for my Linux version and architecture echo \ "deb [arch=$(dpkg --print-architecture) \ signed-by=/etc/apt/keyrings/docker.gpg] \ https://download.docker.com/linux/ubuntu \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  6. Update the packet information of all configured repositories via sudo apt update
  7. Install the docker packages via sudo apt install -y \ docker-ce \ docker-ce-cli \ containerd.io \ docker-buildx-plugin \ docker-compose-plugin

Docker container setup

  1. Clone the repository https://github.com/Akros-STM/mlops via git in the wsl filesystem, e.g., under /home/{USER}.
  2. Checkout the branch develop.
  3. Delete the folder .vitualenvs in /home/{USER} if it exists.
  4. Navigate to the root directory of your project /home/{USER}/mlops.
  5. Build the docker image via docker build --build-arg TORCH_VARIANT=cpu -t mlops-cpu . if you do not have a GPU available. If you have a GPU available, then execute the command docker build --build-arg TORCH_VARIANT=cu121 -t mlops-gpu.
  6. Run the docker container via docker run -it --rm -v $(pwd):/workspace -p 8888:8888 -p 8000:8000 -p 5678:5678 mlops-cpu if you do not have a GPU available. If you have a GPU available, run the command docker run -it --rm -v $(pwd):/workspace -p 8888:8888 -p 8000:8000 -p 5678:5678 mlops-gpu to run the container.
  7. To connect your Pycharm environment to the docker container, go to the Pycharm Settings, navigate to "Build, Execution, Deployment" and select the Docker section. Then select your Docker instance and connect to Docker.
  8. Go to the Pycharm Settings, Python, Interpreter, click "Add interpreter" and select "On Docker". Then, "Pull or use existing", enter the name of your docker container, i.e., either mlops-cpu or mlops-gpu depending on whether you have a GPU available.

Explanation of Scripts

Preprocessing

The preprocessing script prepares the NIH ChestX-ray8 (CXR8) dataset from https://nihcc.app.box.com/v/ChestXray-NIHCC for deep learning workflows. It automatically extracts all image .tar archives, loads the NIH metadata CSV, generates multi-label disease targets, and creates patient-level train/validation/test splits to avoid data leakage between splits. All chest X-ray images are converted to grayscale, resized to a configurable resolution (default: 224x224), and stored in structured output folders. Additionally, the script generates a manifest.csv file containing image paths, patient IDs, split assignments, and multi-hot encoded disease labels, which can be directly used in PyTorch or TensorFlow training pipelines for multi-label chest X-ray classification.

End-to-end monitoring workflow

The current reference workflow implements the abstract's core MLOps loop: preprocessing, MLflow-tracked training, image/data drift monitoring, embedding drift, prediction drift, controlled shift evaluation, API deployment, alert generation, and retraining triggers.

Set PYTHONPATH before running the scripts outside Docker:

export PYTHONPATH=/workspace/src

On Windows PowerShell from the repository root:

$env:PYTHONPATH="src"

1. Preprocess CXR8

python src/dataset/dataset_preprocessing.py \
  --data_dir /workspace/data/CXR8-selected \
  --out_dir /workspace/data/cxr8_preprocessed \
  --size 224 \
  --seed 42

This writes resized images into split folders and creates:

/workspace/data/cxr8_preprocessed/manifest.csv

2. Train with MLflow and monitoring artifacts

python src/training/cxr8_training_with_augmentation_weighted_loss_mlflow.py \
  --data_root /workspace/data/cxr8_preprocessed \
  --manifest_csv /workspace/data/cxr8_preprocessed/manifest.csv \
  --out_dir /workspace/data/reports \
  --epochs 10 \
  --batch_size 32 \
  --use_pos_weight \
  --use_mlflow \
  --mlflow_tracking_uri http://localhost:5000 \
  --mlflow_experiment cxr8-monitoring \
  --mlflow_log_model

Outputs include best_model.pt, training_and_drift_report.json, whylogs profiles, Evidently HTML reports, image-feature CSVs, per-class metrics, embedding drift, prediction drift, and model-weight drift.

3. Evaluate controlled distribution shifts

python src/evaluation/shift_evaluation.py \
  --data_root /workspace/data/cxr8_preprocessed \
  --manifest_csv /workspace/data/cxr8_preprocessed/manifest.csv \
  --model_path /workspace/data/reports/best_model.pt \
  --out_dir /workspace/data/shift_eval \
  --split test \
  --shifts brightness contrast noise \
  --severities 0.1 0.2 0.3

This simulates deployment shifts and writes shift_evaluation_summary.csv plus shift_evaluation_report.json.

4. Create small drift-demo datasets

For demonstrations, create small manifests that are not used for hyperparameter tuning. The most useful first experiment is to train on AP images only and monitor PA images as a held-out acquisition-view drift cohort.

python src/dataset/create_drift_demo_manifests.py \
  --manifest_csv /workspace/data/cxr8_preprocessed/manifest.csv \
  --metadata_csv /workspace/data/CXR8-selected/Data_Entry_2017.csv \
  --out_dir /workspace/data/drift_demo_manifests \
  --max_train_rows 5000 \
  --max_eval_rows 1000 \
  --seed 42

Important outputs:

ap_only_training_manifest.csv      # AP-only train/val/test for training an AP-domain model
ap_reference_train_manifest.csv    # AP train reference cohort for monitoring
ap_id_test_manifest.csv            # AP test cohort, in-distribution baseline
pa_view_drift_test_manifest.csv    # PA test cohort, held-out view-position drift
drift_demo_dataset_report.json     # counts and candidate drift cohorts

Train the AP-only model:

python src/training/cxr8_training_with_augmentation_weighted_loss_mlflow.py \
  --data_root /workspace/data/cxr8_preprocessed \
  --manifest_csv /workspace/data/drift_demo_manifests/ap_only_training_manifest.csv \
  --out_dir /workspace/data/reports_ap_only \
  --epochs 10 \
  --batch_size 32 \
  --use_pos_weight \
  --use_mlflow \
  --mlflow_tracking_uri http://localhost:5000 \
  --mlflow_experiment cxr8-ap-pa-drift

Then compare AP test as the in-distribution baseline and PA test as the drift cohort.

Run a combined domain-shift and concept-drift indicator experiment:

python src/evaluation/domain_concept_drift_experiment.py \
  --data_root /workspace/data/cxr8_preprocessed \
  --model_path /workspace/data/reports_ap_only/best_model.pt \
  --reference_manifest /workspace/data/drift_demo_manifests/ap_id_test_manifest.csv \
  --current_manifest /workspace/data/drift_demo_manifests/pa_view_drift_test_manifest.csv \
  --reference_split test \
  --current_split test \
  --out_dir /workspace/data/domain_concept_ap_vs_pa

The report separates domain-shift evidence from concept-drift indicators. Domain shift is measured with image-feature drift, embedding drift, and prediction drift. Concept drift is reported as a set of indicators using labeled data: AUC/AP deltas, Brier score delta, calibration error delta, threshold error rates, and label-prevalence changes.

The experiment also writes an Evidently prediction/target drift report from the model outputs:

prediction_target_drift_report.html
prediction_target_drift_report.reference_predictions.csv
prediction_target_drift_report.current_predictions.csv

Optional Arize logging uses Arize SDK v8 and can be enabled when ARIZE_API_KEY and ARIZE_SPACE_ID are available:

python src/evaluation/domain_concept_drift_experiment.py \
  --data_root /workspace/data/cxr8_preprocessed \
  --model_path /workspace/data/reports_ap_only/best_model.pt \
  --reference_manifest /workspace/data/drift_demo_manifests/ap_id_test_manifest.csv \
  --current_manifest /workspace/data/drift_demo_manifests/pa_view_drift_test_manifest.csv \
  --reference_split test \
  --current_split test \
  --out_dir /workspace/data/domain_concept_ap_vs_pa \
  --enable_arize \
  --arize_space_id "$ARIZE_SPACE_ID" \
  --arize_model_id cxr8-ap-pa-drift \
  --arize_environment validation \
  --arize_region EU_WEST_1A \
  --arize_batch_id ap-vs-pa-demo \
  --arize_max_rows_per_cohort 250

5. Monitor a new batch and generate alerts

python src/monitoring/monitor_batch.py \
  --data_root /workspace/data/cxr8_preprocessed \
  --reference_manifest /workspace/data/drift_demo_manifests/ap_reference_train_manifest.csv \
  --baseline_manifest /workspace/data/drift_demo_manifests/ap_id_test_manifest.csv \
  --current_manifest /workspace/data/drift_demo_manifests/pa_view_drift_test_manifest.csv \
  --reference_split train \
  --baseline_split test \
  --current_split test \
  --model_path /workspace/data/reports_ap_only/best_model.pt \
  --out_dir /workspace/data/monitoring_ap_train_vs_pa_test \
  --trigger_retraining_file /workspace/data/monitoring_ap_train_vs_pa_test/retrain_required.json

This writes monitoring_report.json, alerts.json, monitoring_report.html, and linked Evidently HTML reports into the output directory.

This creates image-feature drift statistics, Evidently reports, embedding drift, prediction drift, performance drift, alerts.json, and a retraining trigger file when a critical threshold is crossed.

When --model_path is provided, batch monitoring also creates an Evidently prediction/target drift report. Add --enable_arize plus Arize credentials to send the same labeled prediction cohorts to Arize.

6. Serve the model

Build the image:

docker build --build-arg TORCH_VARIANT=cpu -t mlops-cpu .

Start MLflow and the inference API:

docker compose up mlflow api

The API exposes:

GET  http://localhost:8000/health
POST http://localhost:8000/predict

Predictions are appended to /workspace/data/prediction_log.csv, which can be used as the basis for future production-batch monitoring.

7. Operate monitoring, retraining, and registry lifecycle

Run a one-shot monitoring job from a JSON config:

python -m operations.monitoring_job \
  --config configs/monitoring_job.example.json \
  --max_runs 1 \
  --history_json /workspace/data/monitoring_job_history.json

Run it as a scheduled container job. The example uses --interval_seconds 86400 for daily monitoring:

docker compose up monitoring-job

The monitoring job can run three steps:

monitoring_command  -> runs batch monitoring
summary_command     -> writes monitoring_summary.md
retraining_command  -> starts retraining when critical alerts exist
rollback_command    -> optional rollback if retraining fails or critical alerts require it

For manual retraining from an alert file:

python -m operations.retraining_orchestrator \
  --alerts_json /workspace/data/monitoring_ap_train_vs_pa_test/alerts.json \
  --status_json /workspace/data/retraining/retraining_status.json \
  --data_root /workspace/data/cxr8_preprocessed \
  --manifest_csv /workspace/data/drift_demo_manifests/ap_only_training_manifest.csv \
  --out_dir /workspace/data/retraining \
  --epochs 10 \
  --batch_size 32 \
  --use_pos_weight \
  --use_mlflow \
  --mlflow_tracking_uri http://localhost:5000 \
  --mlflow_experiment cxr8-retraining \
  --mlflow_log_model \
  --registered_model_name cxr8-densenet121 \
  --promote_to_stage Staging \
  --candidate_validation_json /workspace/data/retraining/candidate_validation_report.json \
  --baseline_training_report /workspace/data/reports_ap_only/training_and_drift_report.json \
  --monitoring_report /workspace/data/monitoring_ap_train_vs_pa_test/monitoring_report.json

Manage the MLflow model lifecycle:

python -m operations.mlflow_registry status \
  --tracking_uri http://localhost:5000 \
  --model_name cxr8-densenet121

python -m operations.mlflow_registry promote \
  --tracking_uri http://localhost:5000 \
  --model_name cxr8-densenet121 \
  --version 1 \
  --stage Production

python -m operations.mlflow_registry rollback \
  --tracking_uri http://localhost:5000 \
  --model_name cxr8-densenet121 \
  --stage Production

Run an explicit promotion gate before moving a candidate model to Production:

python -m operations.candidate_validation \
  --candidate_training_report /workspace/data/retraining/training_and_drift_report.json \
  --baseline_training_report /workspace/data/reports_ap_only/training_and_drift_report.json \
  --monitoring_report /workspace/data/monitoring_ap_train_vs_pa_test/monitoring_report.json \
  --prediction_log_validation_report /workspace/data/prediction_log_validation.json \
  --output_json /workspace/data/retraining/candidate_validation_report.json \
  --min_candidate_auc 0.70 \
  --max_auc_drop_vs_baseline 0.03 \
  --max_critical_alerts 0 \
  --require_prediction_log_valid

Create a human approval record and require it for Production promotion:

python -m operations.mlflow_registry approve \
  --tracking_uri http://localhost:5000 \
  --model_name cxr8-densenet121 \
  --version 2 \
  --stage Production \
  --approver "clinical-reviewer" \
  --validation_report /workspace/data/retraining/candidate_validation_report.json \
  --approval_record /workspace/data/retraining/production_approval.json

python -m operations.mlflow_registry promote \
  --tracking_uri http://localhost:5000 \
  --model_name cxr8-densenet121 \
  --version 2 \
  --stage Production \
  --require_approval \
  --approval_record /workspace/data/retraining/production_approval.json

Validate production prediction logs:

python -m deployment.validate_prediction_log \
  --prediction_log /workspace/data/prediction_log.csv \
  --output_json /workspace/data/prediction_log_validation.json

Serve either a local state-dict checkpoint or the current MLflow Production model:

python -m deployment.serve_model \
  --model_uri models:/cxr8-densenet121/Production \
  --mlflow_tracking_uri http://localhost:5000 \
  --prediction_log /workspace/data/prediction_log.csv

Create a compact Markdown observability summary:

python -m operations.monitoring_summary \
  --monitoring_report /workspace/data/monitoring_ap_train_vs_pa_test/monitoring_report.json \
  --alerts_json /workspace/data/monitoring_ap_train_vs_pa_test/alerts.json \
  --output_md /workspace/data/monitoring_ap_train_vs_pa_test/monitoring_summary.md

Create a central HTML dashboard that combines training metrics, shift evaluation, AP-vs-PA drift, monitoring alerts, and poster figures:

python -m operations.central_dashboard \
  --reports_dir /workspace/data/reports \
  --domain_dir /workspace/data/domain_concept_ap_vs_pa \
  --monitoring_dir /workspace/data/monitoring_ap_train_vs_pa_test \
  --shift_eval_dir /workspace/data/shift_eval \
  --figures_dir /workspace/data/poster_figures \
  --output_html /workspace/data/central_dashboard.html