Skip to content

Troubleshooting

Jump to bottom
Rabieh Fashwall edited this page Apr 14, 2026 · 1 revision

Troubleshooting Guide

Workshop-wide troubleshooting. Each module also has its own "Troubleshooting" section with scenario-specific fixes — check there first, then come back here for cross-cutting issues.

Quick Triage

Symptom Likely Cause First thing to try
command not found: python / kubectl / kind / go Tool not installed or not on PATH Re-run the Module 0 install step for that tool; open a new shell
ModuleNotFoundError Virtual env not active or deps not installed Activate venv, then pip install -r requirements.txt
Port XXXX already in use Previous service still running See Port already in use
Docker commands fail / Cannot connect to the Docker daemon Docker Desktop not running Start Docker Desktop and wait for the whale icon to go steady
kubectl hangs or times out kind cluster not running kind get clusters, re-create if missing
Pods stuck ImagePullBackOff on kind Image not loaded into kind kind load docker-image <name> --name mlops-workshop
Hugging Face download fails / times out Network / proxy Retry, increase timeout, or pre-download
Wiki links don't open Pre-April 2026 wiki had double .md extensions Already fixed; pull the latest wiki

Environment Setup

Python / virtual environment

Activate the venv before every session and before pip install:

# macOS / Linux / WSL
source venv/bin/activate

# Windows (PowerShell)
venv\Scripts\Activate.ps1

# Windows (cmd.exe)
venv\Scripts\activate.bat

If python doesn't work, try python3 (macOS/Linux). If neither exists, re-install Python and tick "Add Python to PATH" on Windows.

After activation, always upgrade pip first:

pip install --upgrade pip

pip install fails on Apple Silicon / Windows

  • Apple Silicon: some ML wheels require Xcode CLI tools → xcode-select --install
  • Windows: install Build Tools for Visual Studio (C++ build tools) if a wheel falls back to source
  • Always prefer pre-built wheels — if torch installs from source you'll be waiting 30+ min

Docker & kind

Docker daemon not reachable

  • macOS/Windows: open Docker Desktop, wait until it reports "Running"
  • WSL 2: ensure Docker Desktop's "Use the WSL 2 based engine" is enabled and your distro is integrated (Settings → Resources → WSL Integration)

kind cluster missing or broken

kind get clusters
kind delete cluster --name mlops-workshop
kind create cluster --config modules/module-0/kind.yaml

ImagePullBackOff in kind

Kind nodes don't share your Docker daemon — you must load local images explicitly:

kind load docker-image <image>:<tag> --name mlops-workshop
kubectl rollout restart deployment/<name>

Disk pressure / cluster evicting pods

Between modules, free space:

docker system prune -a --volumes

Allocate at least 8 GB RAM and 20 GB disk to Docker Desktop (Settings → Resources).

Port already in use

Find what's holding the port, then stop it or use another.

# macOS / Linux
lsof -i :5000
kill -9 <PID>

# Windows (PowerShell)
Get-NetTCPConnection -LocalPort 5000 | Select-Object OwningProcess
Stop-Process -Id <PID> -Force

# Windows (cmd.exe)
netstat -ano | findstr :5000
taskkill /PID <PID> /F

Common workshop ports: 5000 (MLflow), 3000 (BentoML), 8080 (Kubeflow / kubectl port-forward), 9090 (Prometheus), 3001 (Grafana).

If you'd rather run on a different port, most services take --port:

mlflow ui --port 5001

MLflow

  • UI shows nothing: you haven't logged any runs yet; run a training script first.
  • mlflow command not found after install: restart your shell or re-activate your venv.
  • Experiment "Default" unexpected: you forgot mlflow.set_experiment("name") — add it before mlflow.start_run().
  • FileNotFoundError: mlruns: MLflow writes locally by default; make sure you're running from the module's working directory so all runs land in the same mlruns/ folder.

For deeper MLflow issues see Module 1 Troubleshooting.

BentoML

  • bentoml serve fails to find service: you must run it from the directory containing service.py.
  • Pydantic validation errors: BentoML 1.4+ requires Pydantic v2 syntax (@field_validator, not @validator). See Module 2 Troubleshooting.
  • Model not found: make sure the MLflow model URI points to the correct registered model/alias.

Kubernetes

  • Pod stuck Pending: check resource requests — kind has limited CPU/memory. kubectl describe pod <name> will show the scheduler reason.
  • Pod CrashLoopBackOff: kubectl logs <pod> --previous for last crash output.
  • Service not reachable via localhost: use kubectl port-forward svc/<name> <port>:<port> rather than relying on NodePort.
  • HPA stuck at unknown/80%: metrics-server isn't installed on kind by default; see Module 3 Troubleshooting.

Kubeflow Pipelines

Kubeflow install takes 5–10 minutes on first run and is the most fragile step in the workshop.

  • install-kubeflow.sh times out waiting for pods: run kubectl get pods -n kubeflow -w in another terminal and watch for ImagePullBackOff or CrashLoopBackOff.
  • MinIO CrashLoopBackOff: known compatibility issue; the install script applies a patch — re-run it.
  • UI unreachable at localhost:8080: kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80 must stay running in a terminal.

See Module 5 Troubleshooting for component-level fixes.

Monitoring (Prometheus / Grafana)

  • Grafana default login: admin / admin (you'll be prompted to change).
  • No metrics in Prometheus: check /targets page — your service must expose /metrics on the port Prometheus scrapes.
  • See Module 6 Troubleshooting.

CI/CD (GitHub Actions)

  • ghcr.io push denied: workflow needs packages: write permission in the job.
  • Workflow doesn't trigger: check branch filter in on.push.branches.
  • See Module 7.

Windows-specific gotchas

  • .sh scripts won't execute natively. Use WSL 2 Ubuntu terminal (recommended) or Git Bash. Pure cmd.exe / PowerShell without WSL won't work for the workshop's setup scripts.
  • Line endings corrupt shell scripts. If .sh files give bad interpreter: /bin/bash^M, Git converted LF → CRLF. The repo's .gitattributes prevents this for fresh clones; for existing clones run git config core.autocrlf input and re-clone.
  • Path separators. Use forward slashes / in config files (YAML, Python, shell) — they work on Windows too. Only use backslashes in Windows-native terminals.
  • localhost inside WSL. Docker Desktop on Windows bridges localhost between WSL and host — if it's flaky, try http://host.docker.internal from inside containers.

Full Windows setup instructions: Module 0 → Option C: Windows.

Still stuck?

  1. Re-read the module's own "Troubleshooting" section.
  2. git status / git log — make sure you're on the right branch and up to date.
  3. Reset the environment for that module: delete the kind cluster, docker system prune, rebuild.
  4. Ask the facilitator with: the exact command you ran, the full error output, and your OS + Python version.

Clone this wiki locally