Softmax Classifier Implementation

A complete implementation of a Softmax classifier with cross-entropy loss for multi-class classification, applied to the CIFAR-10 dataset.

Overview

This project implements both naive and vectorized versions of the Softmax classifier with the following features:

• Numerical Stability: Proper handling of overflow/underflow issues
• Vectorized Implementation: Efficient matrix operations for faster training
• Cross-entropy Loss: Standard loss function for multi-class classification
• L2 Regularization: Prevents overfitting
• CIFAR-10 Dataset: Real-world image classification task

Features

• ✅ Naive implementation with explicit loops
• ✅ Vectorized implementation for efficiency
• ✅ Numerical stability improvements
• ✅ L2 regularization support
• ✅ Training visualization
• ✅ Model saving/loading
• ✅ Performance comparison between implementations

Installation

1. Clone the repository:

git clone https://github.com/HoomKh/softmax-classifier.git
cd softmax-classifier

2. Install dependencies:

pip install -r requirements.txt

3. Download the CIFAR-10 dataset:
   • Place the cifar-10-batches-py folder in the dataset/ directory
   • Or update the path in the notebook to point to your dataset location

Project Structure

softmax-classifier/
├── README.md
├── requirements.txt
├── softmax.py                 # Core softmax implementations
├── linear_classifier.py       # Linear classifier base class
├── data_utils.py             # Data loading utilities
├── softmax_classifier.ipynb  # Main training notebook
└── dataset/                  # CIFAR-10 dataset (not included)
    └── cifar-10-batches-py/

Usage

Running the Notebook

1. Start Jupyter:

jupyter notebook

2. Open softmax_classifier.ipynb

3. Run all cells to:

   • Load and preprocess CIFAR-10 data
   • Train the softmax classifier
   • Visualize training progress
   • Evaluate model performance

Key Parameters

• Learning Rate: 1e-7 (default)
• Regularization: 2.5e4 (default)
• Training Iterations: 1500 (default)
• Batch Size: 200 (default)

Implementation Details

Softmax Loss Function

The softmax loss combines:

• Softmax activation: P(y=k|x) = exp(s_k) / Σ exp(s_i)
• Cross-entropy loss: L = -log(P(y_true|x))
• L2 regularization: R(W) = λ * Σ W²

Numerical Stability

The implementation includes several numerical stability improvements:

• Subtracting maximum score before exponential
• Proper gradient normalization
• Correct regularization loss calculation

Results

On CIFAR-10 dataset:

• Training Accuracy: ~32.6%
• Validation Accuracy: ~34%
• Training Time: ~0.9 seconds for 1500 iterations

Performance Comparison

Implementation	Training Time	Loss
Naive	~2.5s	2.34
Vectorized	~0.07s	2.34

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contact

• Email: khoshbinhooman@gmail.com
• GitHub: HoomKh

Acknowledgments

• CIFAR-10 dataset from CIFAR
• Implementation based on CS231n course materials

Troubleshooting

Common Issues

1. Module not found errors: Ensure all dependencies are installed
2. Dataset path errors: Update the cifar10_dir path in the notebook
3. Memory issues: Reduce batch size or use smaller dataset subset
4. Numerical instability: The implementation includes fixes for this

Getting Help

If you encounter any issues:

1. Check the troubleshooting section above
2. Search existing issues on GitHub
3. Create a new issue with detailed error information