Gradient Descent: Optimizing Theta Values

A robust implementation of Logistic Regression using Gradient Descent to find optimal θ (theta) values. This project demonstrates the mathematical foundations of machine learning, featuring L2 regularization, convergence monitoring, and decision boundary visualization.

° ° ° °

---

Technical Architecture

The core of this project is a custom-built LogisticRegression class that implements the optimization loop from scratch. Unlike standard library implementations, this codebase provides deep visibility into the convergence process:

1. Iterative Optimization: Uses Gradient Descent to update theta values until the Euclidean distance between successive iterations falls below a predefined epsilon.
2. Regularization (L2): Incorporates a regularization lambda to prevent overfitting by penalizing large theta coefficients.
3. Standardization: Features a preprocessing pipeline that scales input data (Z-score normalization) to ensure stable gradient steps.
4. Convergence Monitoring: Real-time tracking of the cost function values for debugging and performance verification.

---

Project Structure

GradientDescent_Optimizing-Theta-Values/
├── LICENSE                                   # MIT License
├── README.md                                 # Project documentation
├── .gitattributes                            # Git configuration
├── Project Report.pdf                        # IEEE-style technical report
│
└── Code/                                     # Core Implementation
    ├── logreg.py                             # Reusable Logistic Regression class
    ├── test_logreg1.py                       # Main execution and plotting script
    └── data1.dat                             # Sample dataset (features + labels)

---

Mathematical Foundations

1. The Sigmoid Function

Maps any real-valued number into a value between 0 and 1, facilitating probabilistic classification.

hθ(x) = 1 / (1 + e^(-θᵀx))

2. Regularized Cost Function

The objective function minimized via gradient descent, including the L2 penalty term.

J(θ) = -1/n * Σ [y(i)log(hθ(x(i))) + (1-y(i))log(1-hθ(x(i)))] + λ/2n * Σ θⱼ²

3. Gradient Update

The parameter update rule applied at each iteration to converge towards the global minimum.

θⱼ := θⱼ - α * [1/n * Σ (hθ(x(i)) - y(i))xⱼ(i) + λ/n * θⱼ]

---

Technical Specifications

Parameter	Description
Algorithm	Logistic Regression (Regularized)
Optimization	Vanishing Gradient Descent
Learning Rate (α)	Configurable (Default: 0.01)
Regularization (λ)	Configurable (Default: 0.01)
Convergence (ε)	Distance threshold (Default: 0.0001)

---

Deployment & Installation

Repository Acquisition

Clone the repository and navigate to the project directory:

git clone https://github.com/Zer0-Bug/GradientDescent_Optimizing-Theta-Values.git

cd GradientDescent_Optimizing-Theta-Values

Setup & Dependencies

It is recommended to use a virtual environment:

# Create and activate environment
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install requirements
pip install numpy matplotlib

Execution

Run the test script to execute the training process and visualize the decision boundary:

python Code/test_logreg1.py

---

Contribution

Contributions are always appreciated. Open-source projects grow through collaboration, and any improvement—whether a bug fix, new feature, documentation update, or suggestion—is valuable.

To contribute, please follow the steps below:

1. Fork the repository.
2. Create a new branch for your change:
   git checkout -b feature/your-feature-name
3. Commit your changes with a clear and descriptive message:
   git commit -m "Add: brief description of the change"
4. Push your branch to your fork:
   git push origin feature/your-feature-name
5. Open a Pull Request describing the changes made.

All contributions are reviewed before being merged. Please ensure that your changes follow the existing code style and include relevant documentation or tests where applicable.

---

References

1. Ng, A. - CS229 Lecture Notes: Logistic Regression. Stanford University.

---

×

---

∞