I implemented the foundational algorithms of scikit-learn from scratch using only raw Python and NumPy. You can use this repository to understand the core mathematical mechanics behind modern Machine Learning models, learn vectorized array operations, and build a clean, unified object-oriented estimator interface.
This project goes beyond isolated scripts to build a full, installable Python library—from vectorizing closed-form linear algebra and gradient optimization to building recursive decision trees, ensemble methods, and automated unit test comparisons against the real scikit-learn.
Here is the path we will walk, end to end:
- Who this is for
- Prerequisites
- Setup & Installation
- Code Structure
- Step 1: The Unified Estimator API
- Step 2: Linear Models (Closed-Form Math)
- Step 3: Iterative Models & Gradient Descent
- Step 4: Non-Linear Models & Recursive Trees
- Step 5: Distance-Based Models & Preprocessing
- Step 6: Ensemble Methods (Random Forest)
- Step 7: Verification & Comparative Testing
- Progress
- What's Next
- If you are a student: Read top to bottom. Every algorithm section breaks down the raw mathematical formula first, followed by the plain Python/NumPy vectorized implementation.
- If you are a developer: The repo is built as an editable Python package. You can clone it, inspect the modules, run
pytest, and importminilearndirectly into your own scripts. - If you are preparing for technical ML interviews: This repo covers the most frequently asked "implement X from scratch" questions (OLS regression, k-NN broadcasted distance, Gini impurity, decision tree splitting).
You need a basic understanding of Python Object-Oriented Programming (OOP) and introductory linear algebra (matrix multiplication, transposes, inverses).
| Topic | Focus Area | Recommended Resource |
|---|---|---|
| Python OOP | Classes, Inheritance, Method Overriding | Python OOP Tutorial |
| NumPy | Vectorization, Broadcasting, axis operations |
NumPy Visual Guide |
| Linear Algebra | Matrix multiplication ( |
3Blue1Brown Linear Algebra |
Clone the repository and install it in editable mode (-e). This places minilearn on your Python import path so you can import it from anywhere on your system.
git clone https://github.com/your-username/minilearn.git
cd minilearn
pip install -e .
To run the automated comparison tests against official scikit-learn, install the test suite extras:
pip install -e ".[test]"
minilearn/
├── minilearn/ # Core library package
│ ├── __init__.py # Exposes top-level estimators
│ ├── base.py # BaseEstimator and TransformerMixin
│ ├── linear_model/ # Regression & classification models
│ │ ├── _regression.py # LinearRegression, Ridge
│ │ ├── _logistic.py # LogisticRegression (gradient descent)
│ │ └── _lasso.py # Lasso (planned)
│ ├── tree/ # Tree-based models
│ │ ├── _classes.py # DecisionTreeClassifier, DecisionTreeRegressor
│ │ └── _forest.py # RandomForestClassifier (bagging)
│ ├── neighbors/ # Distance-based models
│ │ └── _knn.py # KNeighborsClassifier
│ ├── preprocessing/ # Data transformers
│ │ └── _data.py # StandardScaler, MinMaxScaler
│ ├── model_selection/ # Data splitting utilities
│ │ └── _split.py # train_test_split
│ ├── metrics/ # Evaluation metrics
│ │ ├── _classification.py # accuracy_score
│ │ └── _regression.py # r2_score
│ └── utils/ # Shared helpers
│ └── validation.py # validate_X, validate_X_y, check_is_fitted
├── tests/ # Pytest suite with sklearn comparisons
│ ├── test_linear_model.py
│ ├── test_tree.py
│ ├── test_neighbors.py
│ ├── test_preprocessing.py
│ ├── test_base.py
│ ├── test_split.py
│ └── test_validation.py
├── pyproject.toml
└── README.md
Every model in scikit-learn adheres to a strict Object-Oriented interface. We enforce this using a shared base class in minilearn/base.py. Learned parameters are always stored with a trailing underscore (e.g., self.coef_, self.intercept_).
Input Matrix X (N x D) ──► .fit(X, y) ──► Saves learned weights (self.coef_)
Input Matrix X_test ──► .predict(X) ──► Computes predictions (X_test @ self.coef_)
import numpy as np
class BaseEstimator:
"""Base interface for all minilearn algorithms."""
def fit(self, X, y=None):
raise NotImplementedError("Subclasses must implement fit()")
def predict(self, X):
raise NotImplementedError("Subclasses must implement predict()")Linear Regression fits a linear model with coefficients
Instead of iterative loops, we solve for
To include regularisation (
# minilearn/linear_model/_regression.py
import numpy as np
from ..base import BaseEstimator
class LinearRegression(BaseEstimator):
def __init__(self, fit_intercept=True):
self.fit_intercept = fit_intercept
self.coef_ = None
self.intercept_ = None
def fit(self, X, y):
X = np.asarray(X, dtype=np.float64)
y = np.asarray(y, dtype=np.float64)
# Add bias column of 1s if intercept is requested
if self.fit_intercept:
X_b = np.c_[np.ones((X.shape[0], 1)), X]
else:
X_b = X
# Solve Normal Equation using pseudo-inverse for numerical stability
beta = np.linalg.pinv(X_b.T @ X_b) @ X_b.T @ y
if self.fit_intercept:
self.intercept_ = beta[0]
self.coef_ = beta[1:]
else:
self.intercept_ = 0.0
self.coef_ = beta
return self
def predict(self, X):
X = np.asarray(X, dtype=np.float64)
return X @ self.coef_ + self.intercept_When a closed-form solution doesn't exist (e.g., Logistic Regression), we use iterative numerical optimization.
Binary classification uses the Sigmoid function to convert raw linear outputs into probabilities:
We update parameter weights iteratively using gradient descent over Binary Cross-Entropy loss:
# minilearn/linear_model/_logistic.py
import numpy as np
from ..base import BaseEstimator
class LogisticRegression(BaseEstimator):
def __init__(self, lr=0.01, n_iters=1000):
self.lr = lr
self.n_iters = n_iters
self.weights = None
self.bias = None
def _sigmoid(self, z):
# Clip z to prevent numerical overflow in exp
z = np.clip(z, -500, 500)
return 1 / (1 + np.exp(-z))
def fit(self, X, y):
n_samples, n_features = X.shape
self.weights = np.zeros(n_features)
self.bias = 0.0
for _ in range(self.n_iters):
linear_model = X @ self.weights + self.bias
y_predicted = self._sigmoid(linear_model)
# Compute gradients
dw = (1 / n_samples) * (X.T @ (y_predicted - y))
db = (1 / n_samples) * np.sum(y_predicted - y)
# Update parameters
self.weights -= self.lr * dw
self.bias -= self.lr * db
return self
def predict(self, X):
linear_model = X @ self.weights + self.bias
y_predicted = self._sigmoid(linear_model)
return (y_predicted >= 0.5).astype(int)Decision Trees split data recursively by finding the feature and numerical threshold that minimizes Gini Impurity:
# snippet of split scoring logic in minilearn/tree/_classes.py
def _gini(y):
hist = np.bincount(y)
ps = hist / len(y)
return 1.0 - np.sum(ps ** 2)
def _best_split(X, y):
best_gain = -1
split_idx, split_thresh = None, None
for feat_idx in range(X.shape[1]):
thresholds = np.unique(X[:, feat_idx])
for thresh in thresholds:
# Create boolean split mask
left_mask = X[:, feat_idx] <= thresh
right_mask = ~left_mask
if len(y[left_mask]) == 0 or len(y[right_mask]) == 0:
continue
# Calculate information gain
parent_gini = _gini(y)
n = len(y)
n_l, n_r = len(y[left_mask]), len(y[right_mask])
child_gini = (n_l/n) * _gini(y[left_mask]) + (n_r/n) * _gini(y[right_mask])
ig = parent_gini - child_gini
if ig > best_gain:
best_gain, split_idx, split_thresh = ig, feat_idx, thresh
return split_idx, split_threshDecision trees are built recursively in fit() and traversed in predict(). See minilearn/tree/_classes.py for the full DecisionTreeClassifier and DecisionTreeRegressor implementations.
Some algorithms are highly sensitive to feature scale. KNN uses Euclidean distance, and gradient descent converges faster on normalized features. We add two transformers and a lazy classifier.
fit() stores the training set. predict() computes vectorized squared distances and takes a majority vote among the k closest neighbors:
# minilearn/neighbors/_knn.py
diff = X[:, None, :] - self.X_train_[None, :, :]
distances = np.sum(diff ** 2, axis=2)StandardScaler centers each feature to zero mean and unit variance. MinMaxScaler rescales features to a target range (default [0, 1]):
# minilearn/preprocessing/_data.py
# StandardScaler transform
(X - self.mean_) / self.scale_
# MinMaxScaler transform
(X - self.min_) / self.scale_ * (max_val - min_val) + min_valAlways fit scalers on training data only, then transform both train and test sets.
A Random Forest trains many decision trees on bootstrap samples of the data and aggregates their votes. This reduces variance compared to a single deep tree.
# minilearn/tree/_forest.py
for _ in range(self.n_estimators):
indices = rng.randint(0, n_samples, size=n_samples) # bootstrap sample
tree = DecisionTreeClassifier(max_depth=self.max_depth)
tree.fit(X[indices], y[indices])
self.estimators_.append(tree)At prediction time, each tree votes and the majority class wins.
We use pytest to compare our library outputs against official scikit-learn outputs on identical synthetic datasets generated via sklearn.datasets.
Run the full test suite:
pytest tests/Current coverage includes sklearn comparison tests for:
| Test file | Models covered |
|---|---|
test_linear_model.py |
LinearRegression, Ridge, LogisticRegression |
test_tree.py |
DecisionTreeClassifier, DecisionTreeRegressor, RandomForestClassifier |
test_neighbors.py |
KNeighborsClassifier |
test_preprocessing.py |
StandardScaler, MinMaxScaler |
import pytest
import numpy as np
from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression as SklearnLinearRegression
from minilearn.linear_model import LinearRegression as CustomLinearRegression
def test_linear_regression_against_sklearn():
# 1. Generate identical test dataset
X, y = make_regression(n_samples=500, n_features=8, noise=0.1, random_state=42)
# 2. Fit official scikit-learn model
sk_model = SklearnLinearRegression(fit_intercept=True)
sk_model.fit(X, y)
# 3. Fit custom minilearn model
custom_model = CustomLinearRegression(fit_intercept=True)
custom_model.fit(X, y)
# 4. Verify outputs match within 1e-5 relative tolerance
np.testing.assert_allclose(custom_model.coef_, sk_model.coef_, rtol=1e-5)
np.testing.assert_allclose(custom_model.intercept_, sk_model.intercept_, rtol=1e-5)
np.testing.assert_allclose(custom_model.predict(X), sk_model.predict(X), rtol=1e-5)- BaseEstimator
- RegressorMixin
- ClassifierMixin
- TransformerMixin
- Input validation (validate_X, validate_X_y, check_is_fitted)
- train_test_split
- LinearRegression
- Ridge
- LogisticRegression
- Lasso
- DecisionTreeClassifier
- DecisionTreeRegressor
- RandomForestClassifier
- KNeighborsClassifier
- StandardScaler
- MinMaxScaler
- r2_score
- accuracy_score
- LinearRegression
- Ridge
- LogisticRegression
- DecisionTreeClassifier
- DecisionTreeRegressor
- RandomForestClassifier
- KNeighborsClassifier
- StandardScaler
- MinMaxScaler
- Lasso (coordinate descent)
- Principal Component Analysis (PCA)
- Random Forest feature subsampling (
max_features)
The core supervised learning path is complete. Remaining work:
- Lasso — implement coordinate descent with soft-thresholding in
minilearn/linear_model/_lasso.py - PCA — unsupervised dimensionality reduction via SVD
- Random Forest enhancements — per-split feature subsampling to match sklearn more closely
- More tests — add Lasso and PCA sklearn comparisons once implemented