Deep ML

A repository for my solutions to problems on Deep-ML, a site for LeetCode-style questions for machine learning and data science. For each problem, I decided to use either numpy or pure Python, depending on the type signature of the method, i.e. if the method takes in 2 np.arrays, then I use numpy, else Python.

Collections

Note

Collections have duplicate questions.

1. AlexNet
   • Implement ReLU Activation Function
   • Simple Convolutional 2D Layer
   • Overlapping Max Pooling
   • PCA Colour Augmentation
   • Dropout Layer
2. Attention is All You Need
   • Implement Self-Attention Mechanism
   • Implement Multi-Head Attention
   • Implement Masked Self-Attention
   • Implement Layer Normalization for Sequence Data
   • Positional Encoding Calculator
3. Deep Learning
   1. Linear Algebra
      • Matrix-Vector Dot Product
      • Transpose of a Matrix
      • Dot Product Calculator
      • Scalar Multiplication of a Matrix
      • Calculate Cosine Similarity Between Vectors
      • Calculate Mean by Row or Column
      • Calculate Eigenvalues of a Matrix
      • Calculate 2x2 Matrix Inverse
      • Matrix time Matrix
   2. Probability and Statistics
      • Poisson Distribution Probability Calculator
      • Binomial Distribution Probability
      • Normal Distribution PDF Calculator
      • Descriptive Statistics Calculator
      • Calculate Covariance Matrix
   3. Optimization Techniques
      • Linear Regression Using Gradient Descent
      • Implement Gradient Descent Variants with MSE Loss
      • Implement Adam Optimization Algorithm
      • Implement Lasso Regression using ISTA
   4. Fundamentals of Neural Networks
      • Softmax Activation Function Implementation
      • Implementation of Log Softmax Function
      • Sigmoid Activation Function
      • Implement ReLU Activation Function
      • Leaky ReLU Activation Function
      • Implement the PReLU Activation Function
      • Single Neuron
      • Implementing a Simple RNN
      • Implement a Long Short-Term Memory (LSTM) Network
      • Simple Convolutional 2D Layer
      • GPT-2 Text Generation
   5. Backpropagation
      • Single Neuron with Backpropagation
      • Implementing Basic Autograd Operations
      • Implement a Simple RNN with Backpropagation Through Time (BPTT)
   6. LLM
      • Implement Self-Attention Mechanism
      • The Pattern Weaver's Code
      • Positional Encoding Calculator
      • Implement Multi-Head Attention
      • GPT-2 Text Generation
4. DeepSeek R1
   • Implement the GRPO Objective Function
   • Group Relative Advantage for GRPO
   • KL Divergence Estimator for GRPO
   • Pass@k and Majority Voting Evaluation Metrics
   • Knowledge Distillation Loss
5. DenseNet
   • Single Neuron with Backpropagation
   • Simple Convolutional 2D Layer
   • Implement ReLU Activation Function
   • Implement a Simple Residual Block with Shortcut Connection
   • Implement Global Average Pooling
   • Implement Batch Normalization for BCHW Input
   • Implement a Batch Dense Block with 2D Convolutions
6. GPT 243
   1. Autograd Engine (Value Class & Backpropagation)
      • Implementing Basic Autograd Operations
      • Single Neuron with Backpropagation
   2. Lab: Autograd
      • MNIST: Classification Loss (with Gradient)
   3. Tokenization & Embeddings
      • Character-Level Tokenizer (stoi/itos/BOS)
      • Learned Positional Embeddings
   4. Lab: Tokenization
      • Build a Tokenizer for Language Modelling
   5. Core Building Blocks (Linear, Softmax, RMSNorm)
      • Matrix-Vector Dot Product
      • Softmax Activation Function Implementation
      • Implement RMSNorm (Root Mean Square Layer Normalization)
   6. Lab: Build a Neural Network from Scratch
      • MNIST: Build Neural NEtwork from Scratch (numpy Only)
   7. Multi-Head Attention & KV Cache
      • Implement Self-Attention Mechanism
      • Implement Masked Self-Attention
      • Implement Multi-Head Attention
      • KV Cache for Efficient Autogregressive Attention
   8. Lab: Attention
      • Design Your Own Attention Mechanism
   9. Transformer Block (Residuals, MLP, Activations)
      • Implement a Simple Residual Block with Shortcut Connection
      • Implement Position-wise Feed-Forwards Block with Residual and Dropout
      • Implement the Square ReLU Activation Function
   10. Lab: Activation Function
       • Design Your Own Activation Function
   11. Loss Functions & Cross-Entropy
       • Compute Multi-class Cross-Entropy Loss
       • Implementation of Log Softmax Function
   12. Adam Optimizer & Learning Rate Schedule
       • Implement Adam Optimization Algorithm
       • Linear Learning Rate Decay
   13. Lab: Optimizer
       • Design Your Own Optimizer (numpy)
   14. Training Loop (Putting It All Together)
       • Calculate Number of Parameters in Neural Network
   15. Lab: Full Training Loop
       • Build a Digit Classifier from Scratch
   16. Inference & Text Generation
       • Temperature Sampling
7. LLM Evaluation Methods
   1. Multiple Choice Benchmarks
      • MMLU Letter-Matching Evaluation
      • MMLU Log-Probability Scoring
   2. Verifier-Based Evaluation
      • Boxed Answer Extraction for Math Benchmarks
      • Math Answer Verification with Equivalence Checking
      • Code Execution Verifier for Programming Benchmarks
   3. Preference Leaderboards
      • Elo Rating System for Model Comparison
      • Bradley-Terry Model for Pairwise Rankings
   4. LLM-as-a-Judge
      • Rubric-Based LLM Judge Evaluation
      • Pairwise Preference Judge for LLM Comparison
   5. Other Measures Mentioned in the Post
      • BLEU Score for Text Generation
      • Calculate PReplexity for Language Models
      • Compute Multi-class Cross-Entropy Loss
8. Linear Algebra
   1. Vector Spaces
      • Matrix-Vector Dot Product
      • Transpose of a Matrix
      • Convert Vector to Diagonal Matrix
      • Dot Product Calculator
      • Find the Column Space of a Matrix
      • Calculate Cosine Similarity Between Vectors
   2. Matrix Operations
      • Reshape Matrix
      • Scalar Multiplication of a Matrix
      • Implement Compressed Row Sparse Matrix (CSR) Format Conversion
      • Implement Orthogonal Projection of a Vector onto a Line
      • Implement Compressed Column Sparse Matrix Format (CSC)
      • Transformation Matrix from Basis B to C
      • Matrix Transformation
      • Calculate 2x2 Matrix Inverse
      • Matrix times Matrix
      • Implement Reduced Row Echelon Form (RREF) Function
   3. Eigenvalues and Eigenvectors
      • Calculate Eigenvalues of a Matrix
      • Solve Linear Equations using Jacobi Method
      • Principal Component Analysis (PCA) Implementation
      • SVD of a 2x2 Matrix
   4. Matrix Factorization and Decomposition
      • 2D Translation of Matrix Implementation
      • Gauss-Seidel Method for Solving Linear Systems
      • Singular Value Decomposition (SVD) of 2x2 Matrix
      • Determinant of a 4x4 Matrix using Laplace's Expansion
9. Machine Learning
   1. Linear Algebra
      • Matrix-Vector Dot Product
      • Transpose of a Matrix
      • Dot Product Calculator
      • Scalar Multiplication of a Matrix
      • Calculate Cosine Similarity Between Vectors
      • Calculate Mean by Row or Column
      • Calculate Eigenvalues of a Matrix
      • Calculate 2x2 Matrix Inverse
      • Matrix time Matrix
   2. Probability and Statistics
      • Poisson Distribution Probability Calculator
      • Binomial Distribution Probability
      • Normal Distribution PDF Calculator
      • Descriptive Statistics Calculator
      • Calculate Covariance Matrix
   3. Optimization
      • Linear Regression Using Gradient Descent
      • Implement Gradient Descent Variants with MSE Loss
      • Implement Adam Optimization Algorithm
      • Implement Lasso Regression using ISTA
   4. Model Evaluation
      • Generate a Confusion Matrix for Binary Classification
      • Calculate Accuracy Score
      • Implement Precision Metric
      • Implement Recall Metric in Binary Classification
      • Implement F-Score Calculation for Binary Classification
      • Calculate R-squared for Regression Analysis
      • Calculate Mean Absolute Error (MAE)
      • Calculate Root Mean Square Error (RMSE)
      • Implement K-Fold Cross-Validation
      • Calculate Performance Metrics for a Classification Model
      • Implementation of Log Softmax Function
      • Implement ReLU Activation Function
   5. Classification & Regression Techniques
      • Linear Regression Using Normal Equation
      • Linear Regression Using Gradient Descent
      • Binary Classification with Logistic Regression
      • Calculate Jaccard Index for Binary Classification
      • Pegasos Kernel SVM Implementation
      • Implement AdaBoost Fit Method
      • Softmax Activation Function Implementation
   6. Unsupervised Learning
      • KL Divergence Between Two Normal Distributions
      • Principal Component Analysis (PCA) Implementation
      • K-Means Clustering
   7. Deep Learning
      • Single Neuron
      • Sigmoid Activation Function Understanding
      • Softmax Activation Function Implementation
      • Implementation of Log Softmax
      • Implement ReLU Activation Function
      • Simple Convolutional 2d Layer
      • Implementing a Simple RNN
10. Metadata Normalization (MDN)
    1. Mathematical Prerequisites
       • Linear Regression Using Normal Equation
       • Implement Orthogonal Projection of a Vector onto a Line
    2. Normalization Baselines (What MDN Improves Upon)
       • Implement Batch Normalization for BCHW Input
       • Implement Group Normalization
    3. Core MDN Concepts
       • Implement Code MDN Residualization
       • Distance Correlation for Measuring Metadata Dependence
    4. Advanced MDN (Handling Confounding)
       • MDN with Label Collinearity Control
    5. Lab
       • Feature Deconfounder for Biased Image Data
11. ResNet
    • Single Neuron with Backpropagation
    • Simple Convolutional 2D Layer
    • Implement ReLU Activation Function
    • Implement a Simple Residual Block with Shortcut Connection
    • Implement Global Average Pooling
    • Implement Batch Normalization for BCHW Input
12. Sparsely Gated MoE
    • Softmax Activation Function Implementation
    • Single Neuron
    • Calculate Computational Efficiency of MoE
    • Implement Noisy Top-K Gating Function
    • Implement a Sparse Mixture of Experts Layer
13. Data Science I Interview Prep
    1. Core Machine Learning Concepts
       • Linear Regression Using Gradient Descent
       • K-Means Clustering
       • Implement Early Stopping Based on Validation Loss
       • Find the Best Gini-Based Split for a Binary Decision Tree
       • Implement K-Nearest Neighbours
    2. Data Processing
       • One-Hot Encoding of Nominal Values
       • Min-Max Normalization of Feature Values
       • Implement K-Fold Cross-Validation
       • Calculate Mean by Row or Column
       • Feature Scaling Implementation
    3. Deep Learning
       • Dropout Layer
       • Min-Max Normalization of Feature Values
       • Softmax Activation Function Function Implementation
       • Single Neuron
       • Implement ReLU Activation Function
    4. Model Evaluation & Metrics
       • Calculate F1 Score from Predict and True Labels
       • Calculate Accuracy Score
       • Calculate Root Mean Square Error (RMSE)
       • Calculate Mean Absolute Error (MAE)
       • Implement Precision Metric
       • Detect Overfitting or Underfitting
       • ExponentialLR Learning Rate Scheduler
14. Essense of Linear Algebra
    1. Vectors
       • Scalar Multiplication of a Matrix
    2. Linear Combinations
       • Compute Orthonormal Basis for 2D Vectors
    3. Linear Transformations
       • Matrix-Vector Dot Product
       • Dot Product Calculator
       • Transformation Matrix from Basis B to C
    4. Matrix Multiplication
       • Matrix times Matrix
    5. Determinant
       • Determinant of a 4x4 Matrix using Laplace's Expansion
    6. Inverse Matrices
       • Calculate 2x2 Matrix Inverse
       • Implement Reduced Row Echelon Form (RREF) Function
       • Find the Column Space of a Matrix
    7. Cross Product
       • Compute the Cross Product of Two 3D Vectors
    8. Cramer's Rule
       • Solve System of Linear Equations Using Cramer's Rule
    9. Change of Basis
       • Transformation Matrix from Basis B to C
    10. Eigenvector and Eigenvalues
        • Calculate Eigenvalues of a Matrix
15. Micrograd Builder
    • Sigmoid Activation Function Understanding
    • Softmax Activation Function Implementation
    • Implement Basic Autograd Operations
    • Single Neuron
    • Single Neuron with Backpropagation
16. Optimizers
    • Implement Gradient Descent Variants with MSE Loss
    • Implement Adam Optimization Algorithm
    • Adagrad Optimizer
    • Momentum Optimizer
    • Adamax Optimizer
    • Adadelta Optimizer
    • Nesterov Accelerated Gradient Optimizer
    • Find Captain Redbeard's Hidden Treasure

Labs

• MNIST: Pytorch DataLoader
• MNIST: Design-Your-Own tiny Pytorch Model
• MNIST: Design Your Own Pytorch Optimizer
• MNIST: Classification Loss (with Gradient)
• MNIST: Adversarial Example Generation
• MNIST: Build Neural Network from Scratch (numpy Only)
• MNIST: Fix Very Deep Network Training
• Design Your Own Optimizer (numpy)
• Design Your Own Activation Function
• Design Your Own Attention Mechanism
• Data Preprocessing: Handling Missing Values
• PyTorch: Implement Your Own Gradient Descent Training Step
• PyTorch: Build a Complete Training Loop
• Numpy: Design Your Own Dimensionality Reduction
• Dimensionality Reduction with Sklearn
• Feature Deconfounder for Biased Image Data
• Train a Linear Regression Model
• Build a Tokenizer for Language Modeling
• Build a Digit Classifier from Scratch
• Fix Overfitting with Regularization (numpy)
• Fix Overfitting with Regularization (Sklearn)
• Train a Binary Classifier
• Design Your Own Normalization Layer
• Design Your Own MoE Router

Learning Paths

1. Calculus for Machine Learning
   1. Derivatives and Gradients
      • Derivative of a Polynomial
      • Product Rule for Derivatives
      • Quotient Rule for Derivatives
      • Gradient Direction and Magnitude
   2. Multivariate Calculus
      • Partal Derivatives of Multivariable Functions
      • Chain Rule for Composite Functions
      • Jacobian Matrix Calculation
      • Compute the Hessian Matrix
   3. Neural Network Derivatives
      • Derivatives of Activation Functions
      • Derivative of Softmax
      • Derivative of Cross-Entropy Loss w.r.t Logits
   4. Backpropagation
      • Single Neuron with Backpropagation
      • Implementing Basic Autograd Operations
      • Numerical Gradient Checking
   5. Gradient Descent
      • Linear Regression using Gradient Descent
      • Implement Gradient Descent Variants with MSE Loss
      • Taylor Series Approximation
      • Momentum Optimizer
   6. Optimization
      • Find Captain Redbeard's Hidden Treasure
      • Newton's Method for Optimization
      • Classify Critical Points Using Hessian Eigenvalues
      • Lagrange Multipliers for Constrained Quadratic Optimization
   7. Calculus Lab
      • MNIST: Classification Loss (with Gradient)
   8. Pytorch: Calculus Lab 1
      • PyTorch: Implement Your Own Gradient Descent Training Step
   9. Pytorch: Calculus Lab 2
      • PyTorch: Build a Complete Training Loop
2. Linear Algebra for Machine Learning
   1. Vector Operations
      • Dot Product Calculator
      • Vector Element-wise Sum
      • Crompute the Cross Product
      • Calculate Cosine Similarity
   2. Vector Norms and Independence
      • Engram Context-Aware
      • Compute the Null Space of a Matrix
   3. Matrix Basics
      • Transpose of a Matrix
      • Reshape Matrix
      • Scalar Multiplication of Matrix
      • Matrix-Vector Dot Product
   4. Matrix Multiplication
      • Matrix times Matrix
      • Convert Vector to Diagonal Matrix
      • Calculate by Row or Column
   5. Matrix Properties I
      • Matrix Determinant & Trace
      • Calculate 2x2 Matrix Inverse
      • Vector Norms (L1/L2/Frobenius)
   6. Matrix Properties II
      • Calculate Eigenvalues of a Matrix
      • Check Linear Independence of Vectors
      • Matrix Rank
   7. Solving Linear Systems
      • Gaussian Elimination for Solving Linear Systems
      • Implement Reduced Row Echelon Form
      • Solve Linear Equations Using Jacobi Method
      • Gauss-Seidel Method for Solving Linear Systems
   8. Orthogonality and Projections
      • Implement Orthogonal Projection of a Vector Onto a Line
      • Compute Orthonormal Basis for 2D Vectors
   9. Matrix Decompositions I
      • Check if Matrix is Positive Definite
      • LU Decomposition of a Square Matrix
   10. Matrix Decompositions II
       • Singular Value Decomposition of a 2x2 Matrix
       • QR Decomposition
   11. Covariance and Correlation
       • Calculate Covariance Matrix
       • Calculate Correlation Matrix
   12. Linear Algebra Lab I
       • Numpy: Design Your Own Dimensionality Reduction
   13. Linear Algebra Lab II
       • Dimensionality Reduction with Sklearn
3. Probability and Statistics for Machine Learning
   1. Descriptive Statistics
      • Descriptive Statistics Calculator
      • Calculate P50/P95/P99 Latency Percentiles
      • Calculate Covariance Matrix
   2. Probability Fundamentals
      1. • Expected Value and Variance of an n-sided Die
      2. • Empirical Probability Mass Function
      3. • Calculate Conditional Probability
      4. • Compute Total Probability
   3. Bayes' Theorem
   4. Common Distributions I
   5. Common Distributions II
   6. Law of Large Numbers and CLT
   7. Information Theory
   8. KL Divergence
   9. Maximum Likelihood and MAP
   10. Statistical Inference
   11. Bayesian Methods
   12. Probabalistic Models