DSLR

Data Science Logistic Regression — A 42 school project to classify Hogwarts students into their house using logistic regression and exploratory data analysis.

Note

This project combines data exploration, visualization, and machine learning to predict student houses from course grades using custom-built logistic regression with multiple gradient descent variants.

Table of Contents

• Overview
• Feature Highlights
• Architecture
• Getting Started
  • Prerequisites
  • Install Dependencies
  • Dataset Format
  • Run the Pipeline
• Scripts & Usage
  • Data Exploration
  • Model Training
  • Predictions
  • Visualization
• Project Structure
• Further Reading

Overview

DSLR is a end-to-end machine learning project that classifies students into Hogwarts houses (Gryffindor, Slytherin, Ravenclaw, Hufflepuff) using their course grades. The pipeline includes:

1. Data Exploration — Compute and display descriptive statistics (mean, std, percentiles, outliers).
2. Visualization — Generate scatter plots, histograms, and pair plots to explore feature distributions and correlations.
3. Feature Normalization — Standardize features and handle missing values for model training.
4. Model Training — Implement logistic regression with three gradient descent strategies: batch, stochastic, and mini-batch.
5. Predictions — Load trained weights and classify new students into their houses.

The entire system uses hand-written statistical functions (no NumPy-based shortcuts for stats) and a custom logistic regression implementation to reinforce ML fundamentals.

Feature Highlights

• Custom Statistics — Implement mean, standard deviation, percentiles, median, mode, and outlier detection without relying on built-in statistical libraries.
• Multiple Gradient Descent Methods — Batch GD, stochastic GD, and mini-batch GD with early stopping via rolling-window loss tracking.
• Flexible Training — Train with --gd-type batch|stochastic|mini-batch to compare convergence and accuracy.
• Comprehensive Visualization — Scatter plots, histograms, and pair plots to identify feature importance and inter-house patterns.
• One-vs-Rest Classification — Train independent binary logistic regression models for each house and use argmax for multi-class predictions.
• Normalized Pipeline — Handle missing values (fill with median), normalize features, and persist normalization stats for test-time consistency.
• Bonus Statistics — Optional --bonus flag in describe.py shows NaN counts, ranges, modes, and outlier counts per feature.

Architecture

dslr/
├── describe.py              # Print descriptive statistics for the dataset
├── histogram.py             # Plot histograms of scores by house
├── scatter_plot.py          # Plot two features colored by house
├── pair_plot.py             # Grid of scatter plots and histograms (pair-wise)
├── logreg_train.py          # Train logistic regression models
├── logreg_predict.py        # Load weights and predict houses for new data
├── utils.py                 # Shared utilities (stats, data loading, normalization)
├── datasets/
│   ├── dataset_train.csv    # Training data (with Hogwarts House column)
│   └── dataset_test.csv     # Test data (no house labels)
├── requirements.txt         # Python dependencies
├── LICENSE
└── README.md                # This file

Getting Started

Prerequisites

• Python 3.8+
• pip (or your preferred package manager)

Install Dependencies

pip install -r requirements.txt

The project uses:

• pandas — data manipulation
• numpy — numerical computing
• matplotlib — visualization
• scikit-learn — accuracy scoring during training
• tabulate — pretty-printed statistics tables

Dataset Format

Both training and test datasets must be CSV files with:

• A Hogwarts House column (training only)
• 13 course features:
  • Arithmancy, Astronomy, Herbology, Defense Against the Dark Arts
  • Divination, Muggle Studies, Ancient Runes, History of Magic
  • Transfiguration, Potions, Care of Magical Creatures, Charms, Flying

Missing values are represented as nan. The trainer fills them with the median; the predictor mirrors this approach.

Run the Pipeline

1. Explore the data

# Basic statistics
python describe.py datasets/dataset_train.csv

# Include bonus stats (NaN count, range, mode, outliers)
python describe.py datasets/dataset_train.csv --bonus

2. Visualize features

# Single scatter plot (default: Defense Against the Dark Arts vs Astronomy)
python scatter_plot.py datasets/dataset_train.csv

# Scatter plot for two specific features
python scatter_plot.py datasets/dataset_train.csv --subject1 "Potions" --subject2 "Charms"

# Histograms for a single course
python histogram.py datasets/dataset_train.csv --course "Potions"

# Histograms for all 13 courses (4×4 grid with legend)
python histogram.py datasets/dataset_train.csv

# Pair-wise grid: scatter plots (lower triangle) and histograms (diagonal)
python pair_plot.py datasets/dataset_train.csv

3. Train the model

Training produces a weights.json file with learned weights and normalization stats.

# Batch gradient descent (full batch per epoch)
python logreg_train.py datasets/dataset_train.csv

# Stochastic gradient descent (one sample per update)
python logreg_train.py datasets/dataset_train.csv --gd-type stochastic

# Mini-batch gradient descent (32 samples per batch by default)
python logreg_train.py datasets/dataset_train.csv --gd-type mini-batch

Output:

• Training progress logged every 100 epochs
• Final accuracy on training data
• weights.json saved to the current directory

4. Make predictions

Load the trained model and predict houses for a test set:

python logreg_predict.py datasets/dataset_test.csv

Output:

• houses.csv with columns: Index, Hogwarts House

Scripts & Usage

Data Exploration

describe.py

Compute and display basic statistics (count, mean, std, min, 25%, 50%, 75%, max) for all 13 features.

python describe.py <data_path> [--bonus]

• <data_path> — path to the CSV file
• --bonus — also show NaN count, range, mode, and number of outliers per feature

Output: Pretty table with statistics per feature.

---

Visualization

scatter_plot.py

Plot two features (x vs y) with points colored by house.

python scatter_plot.py <data_path> [--subject1 <first>] [--subject2 <second>]

• <data_path> — path to the CSV file
• --subject1 — feature for x-axis (default: "Defense Against the Dark Arts")
• --subject2 — feature for y-axis (default: "Astronomy")

histogram.py

Plot feature distributions (one or all courses) as histograms with house-specific colors.

python histogram.py <data_path> [--course <feature>]

• <data_path> — path to the CSV file
• --course — single course name; omit to plot all 13 features in a 4×4 grid

pair_plot.py

Generate a 13×13 grid of plots:

• Diagonal: histograms (one feature)
• Lower triangle: scatter plots (two features)
• Upper triangle: hidden to avoid redundancy

python pair_plot.py <data_path>

• <data_path> — path to the CSV file

---

Model Training

logreg_train.py

Train one-vs-rest logistic regression models (one per house) and save weights and normalization stats.

python logreg_train.py <train_data> [--gd-type {batch,stochastic,mini-batch}]

• <train_data> — path to the training CSV
• --gd-type — gradient descent method (default: "batch")
  • batch — full-batch GD
  • stochastic — online GD (one sample per update)
  • mini-batch — mini-batch GD (32 samples per batch)

Output:

• Per-house training progress and final accuracy
• weights.json containing:
  • features — list of 8 training features (14 total minus 5 excluded)
  • means and stds — normalization parameters per feature
  • weights — learned coefficients (including bias) per house model

---

Predictions

logreg_predict.py

Load trained weights and predict houses for a test set.

python logreg_predict.py <test_data>

• <test_data> — path to the test CSV (no "Hogwarts House" column needed)

Output:

• houses.csv with columns Index, Hogwarts House

The script:

1. Loads the test data
2. Loads weights.json (normalization stats and model weights)
3. Normalizes test features using stored means and stds
4. Computes sigmoid probabilities for each house
5. Selects the house with the highest probability per student
6. Writes predictions to houses.csv

---

Project Structure

dslr/
├── describe.py                 # Descriptive statistics (mean, std, percentiles, etc.)
├── histogram.py                # Histogram visualization by house
├── scatter_plot.py             # Scatter plot (two features, colored by house)
├── pair_plot.py                # Pair-wise grid (histograms + scatter plots)
├── logreg_train.py             # Train logistic regression models
├── logreg_predict.py           # Predict houses using trained weights
├── utils.py                    # Shared utilities
│   ├── HOUSES                  # Tuple of house names
│   ├── FEATURES                # All 13 course features
│   ├── TRAIN_FEATURES          # Subset of 8 features used for training
│   ├── load_data()             # Load and validate CSV
│   ├── score_by_house()        # Group students by house
│   ├── load_weights()          # Load JSON weights file
│   ├── sigmoid()               # Sigmoid activation
│   ├── ft_*()                  # Custom statistics (mean, std, median, etc.)
│   ├── normalize()             # Standardize features
│   └── normalize_features()    # Full normalization pipeline
├── datasets/
│   ├── dataset_train.csv       # Training data
│   └── dataset_test.csv        # Test data
├── requirements.txt            # Python dependencies
├── LICENSE
└── README.md                   # This file

Further Reading

• Logistic Regression — The core model uses a one-vs-rest approach: each house gets its own binary classifier that predicts "this house" vs "not this house." At test time, we pick the house with the highest probability.
• Gradient Descent — Three variants are implemented:
  • Batch updates on the full dataset per epoch (most stable but slow).
  • Stochastic updates on each sample (noisy but fast, may require more epochs).
  • Mini-batch uses a fixed batch size (good balance).
• Early Stopping — Training terminates when the rolling average loss (last 100 epochs) stops improving by a threshold (patience=10, min_delta=1e-4).
• Feature Engineering — The 5 excluded features (Arithmancy, Care of Magical Creatures, Transfiguration, History of Magic, Muggle Studies) were deemed less useful for classification during the design phase.
• Data Normalization — Missing values are filled with the median (computed on training data); features are standardized with z-score normalization (z = (x - mean) / std).

Contributors

This project was developed in collaboration with Cedric.

For questions or improvements, feel free to open an issue or contribute. Happy classifying! 🧙‍♂️