Quantifying the Efficiency-Geometry-Similarity Surface in Sparse Autoencoders Trained on Natural Images

A 12-week reproducible research pipeline producing a publishable paper at the Journal of Computational Neuroscience.

---

Table of Contents

1. Project Overview
2. Repository Structure
3. Environment Setup
4. Step-by-Step Execution Guide
5. Expected Outputs Per Week
6. Troubleshooting
7. Citation

---

Project Overview

This project provides the first systematic, quantitative measurement of the efficiency-geometry-similarity surface across the full sparsity spectrum of sparse autoencoders trained on natural image patches.

Core scientific claim: We quantify how Gabor-filter emergence (Gabor-fit R²), dictionary geometric diversity, and operational efficiency co-vary as a function of L1 sparsity penalty (λ), producing a joint Pareto surface that tests the efficient coding hypothesis with unprecedented specificity.

Target journal: Journal of Computational Neuroscience (JCNS) — Springer Nature
Preprint: bioRxiv (submit simultaneously with journal)

---

Repository Structure

sparse-ae-research/
├── README.md                        ← This file
├── requirements.txt                 ← Pinned Python dependencies
├── reproduce.py                     ← End-to-end pipeline runner
│
├── notebooks/
│   ├── 01_preprocessing.ipynb       ← Week 1: ZCA whitening pipeline
│   ├── 02_sparse_autoencoder.ipynb  ← Week 2: SAE training & validation
│   ├── 03_gabor_fit.ipynb           ← Week 3: Gabor-fit metric instrument
│   ├── 04_metrics_validation.ipynb  ← Week 4: Geometry & efficiency metrics
│   ├── 05_lambda_sweep.ipynb        ← Week 5: Core 8×3 sweep experiment
│   ├── 06_figures.ipynb             ← Week 6: Publication figures
│   ├── 07_pca_baseline.ipynb        ← Week 7: PCA control condition
│   └── 09_methods_results_draft.md  ← Week 9: Writing scaffold
│
├── src/
│   ├── __init__.py
│   ├── gabor_utils.py               ← Gabor-fit functions (importable)
│   ├── metrics_utils.py             ← All three metric functions (importable)
│   └── model.py                     ← SparseAutoencoder class
│
├── data/
│   └── placeholder/                 ← patches_zca.npy goes here (from Google Drive)
│
├── figures/
│   ├── fig1_pareto_frontier.pdf
│   ├── fig2_gabor_r2_curve.pdf
│   └── fig3_four_panel.pdf
│
├── paper/
│   └── manuscript.docx              ← JCNS-formatted manuscript template
│
└── scripts/
    └── validate_environment.py      ← Checks all dependencies before running

---

Environment Setup

Option A — Google Colab (Recommended)

Each notebook is self-contained and mounts Google Drive. No local setup required.

1. Upload this repository to Google Drive or clone it in a Colab cell:

!git clone https://github.com/YOUR_USERNAME/sparse-ae-research.git
%cd sparse-ae-research
!pip install -r requirements.txt

2. Mount your Drive at the top of every notebook:

from google.colab import drive
drive.mount('/content/drive')
RESULTS_DIR = '/content/drive/MyDrive/sparse_ae_results'
import os; os.makedirs(RESULTS_DIR, exist_ok=True)

Option B — Local Environment

git clone https://github.com/YOUR_USERNAME/sparse-ae-research.git
cd sparse-ae-research
python -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate
pip install -r requirements.txt
python scripts/validate_environment.py

---

Step-by-Step Execution Guide

Critical rule: Do not skip ahead. Each week produces artifacts that the next week depends on. The order is mandatory.

---

WEEK 1 — Dataset Pipeline and Whitening Validation

Notebook: notebooks/01_preprocessing.ipynb
Runtime: ~20 min on Colab (CPU is fine)

What to run (in order within the notebook):

Cell 1 — Imports and Drive mount

# Run this cell first; it mounts your Drive
from google.colab import drive
drive.mount('/content/drive')

Cell 2 — Dataset download

# Choose ONE: van Hateren (preferred) or STL-10 fallback
# Option A — van Hateren:
# Download manually from https://bethgelab.org/datasets/vanhateren/
# Upload .iml files to Drive, then load here

# Option B — STL-10 (easier, fully automated):
import torchvision
stl10 = torchvision.datasets.STL10(root='/content/data', split='unlabeled', download=True)

Cell 3 — Patch extraction

# Extracts 100,000 random 16×16 patches; filters blank patches (std < 0.01)
# Run as-is — no parameters to change

Cell 4 — ZCA whitening

# Implements W = E @ diag(1/sqrt(d + ε)) @ E.T with ε=0.1
# Run as-is

Cell 5 — MANDATORY VALIDATION (do not skip)

cov = np.cov(whitened_patches.T)
print(f"Diagonal mean: {np.diag(cov).mean():.4f}")    # Must be ~1.0
print(f"Off-diag std: {cov[~np.eye(256,dtype=bool)].std():.4f}")  # Must be <0.05
assert np.abs(np.diag(cov).mean() - 1.0) < 0.1, "Whitening failed — see README Troubleshooting"

✅ Week 1 gate: Assert must pass before proceeding.
📁 Save: patches_zca.npy → Google Drive RESULTS_DIR

---

WEEK 2 — Sparse Autoencoder: First Convergence

Notebook: notebooks/02_sparse_autoencoder.ipynb
Runtime: ~45 min on Colab GPU (T4 recommended)

Setup: Enable GPU in Colab

Runtime → Change runtime type → T4 GPU

Cell 1 — Load data

# Loads patches_zca.npy from Drive — must exist from Week 1
patches = np.load(f'{RESULTS_DIR}/patches_zca.npy')

Cell 2 — Define model

# Uses SparseAutoencoder from src/model.py
# Architecture: Linear(256→512) + ReLU, tied weights
# Do not modify architecture at this stage

Cell 3 — Train at λ=0.01 (anchor point)

# Adam lr=1e-3, batch=256, 50 epochs
# Logs MSE, sparsity, dead unit count per epoch

Cell 4 — Dead unit check (MANDATORY)

# Must pass: dead_units < 50 of 512
# If fail: see Week 2 troubleshooting below

Cell 5 — Visualize dictionary

# Plots all 512 basis functions as 16×16 patches
# Healthy dictionary shows oriented edge-like structures

✅ Week 2 gate: Dead units < 50/512, MSE < 0.05, sparsity 5–30%.
📁 Save: 02_model_lambda001.pt → Drive

---

WEEK 3 — Gabor-Fit Metric: Building Your Primary Instrument

Notebook: notebooks/03_gabor_fit.ipynb
Runtime: ~30 min (CPU fine)

Cell 1 — Implement gabor_2d function

# 2D Gabor with 8 parameters: (x0, y0, σ, f, θ, φ)
# Code is pre-written in src/gabor_utils.py — import and test here
from src.gabor_utils import gabor_fit_r2

Cell 2 — Synthetic Gabor validation

# Generate known Gabor → fit → check R² > 0.95
# Check parameter recovery within 5% of ground truth
# MUST PASS before proceeding

Cell 3 — Noise filter validation

# Generate 10 random 16×16 arrays → fit → check mean R² < 0.3
# MUST PASS before proceeding

Cell 4 — Apply to Week 2 dictionary

# Fits Gabors to all 512 atoms from Week 2 model
# Saves R² scores per atom

Cell 5 — Export module

# Copies validated gabor_utils.py to Drive for import in Week 5

✅ Week 3 gate: Synthetic R² > 0.95 AND noise mean R² < 0.3.
📁 Save: gabor_utils.py, 03_validation_report.txt → Drive

---

WEEK 4 — Geometry Metric and Efficiency Proxy

Notebook: notebooks/04_metrics_validation.ipynb
Runtime: ~20 min (CPU fine)

Cell 1 — Geometric spread

# Computes mean pairwise cosine distance across 512 atoms
# Uses 10K sampled pairs for memory efficiency
from src.metrics_utils import geometric_spread

Cell 2 — Validate geometric spread ordering

# Random weight matrix must give HIGHER spread than identity matrix
# MUST PASS before proceeding

Cell 3 — Train λ=0 baseline (needed for efficiency normalization)

# Pure autoencoder (no sparsity), 10 epochs only
# Save MSE on held-out 20K patches as normalization denominator

Cell 4 — Efficiency proxy

# efficiency = (1 - mse_normalized) / mean_active_units_per_patch
# mse_normalized = mse(λ) / mse(λ=0)
from src.metrics_utils import efficiency_proxy

Cell 5 — Validate efficiency ordering (TWO TESTS)

# Test A: Lower MSE at same sparsity → higher efficiency (must pass)
# Test B: Same MSE at lower sparsity → higher efficiency (must pass)

Cell 6 — Baseline measurement

# Apply all three metrics to Week 2 model
# Record: Gabor R² mean ± std, geometric spread mean ± std, efficiency proxy
# These are your ANCHOR NUMBERS for comparison

✅ Week 4 gate: Both ordering tests pass. Baseline numbers recorded.
📁 Save: metrics_utils.py, 04_metrics_baseline.txt → Drive

---

WEEK 5 — The Sweep: Core Experiment (POINT OF NO RETURN)

Notebook: notebooks/05_lambda_sweep.ipynb
Runtime: 6–8 hours total (split across 2 Colab sessions)

⚠️ WARNING: After this week your data is fixed. Do not modify architecture, preprocessing, or metrics after running the sweep. Changing anything means restarting from Week 2.

Before starting — verify these exist on Drive:

import os
assert os.path.exists(f'{RESULTS_DIR}/patches_zca.npy'), "Missing Week 1 output"
assert os.path.exists(f'{RESULTS_DIR}/02_model_lambda001.pt'), "Missing Week 2 output"
assert os.path.exists(f'{RESULTS_DIR}/gabor_utils.py'), "Missing Week 3 output"
assert os.path.exists(f'{RESULTS_DIR}/metrics_utils.py'), "Missing Week 4 output"
print("All prerequisites found — safe to sweep")

Cell 1 — Define sweep parameters

LAMBDAS = [0.0, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0]
SEEDS   = [42, 123, 7]
# 8 × 3 = 24 total models

Cell 2 — Run checkpoint-safe sweep loop

# The loop SKIPS already-completed runs (safe to restart)
# Appends each result to sweep_results.csv immediately after training
# Never holds results only in memory
# Expected: ~15–20 min per model on T4 GPU

Resuming after Colab disconnect:

1. Reconnect runtime
2. Re-mount Drive
3. Re-run Cell 1 (imports)
4. Re-run Cell 2 (loop skips completed runs automatically)

Cell 3 — Validation plot

# Quick unstyled plot: efficiency proxy vs λ
# Purpose: confirm data is non-constant before Week 6
# If all efficiency values are identical → metric bug (see Troubleshooting)

✅ Week 5 gate: sweep_results.csv has exactly 24 rows, efficiency proxy varies across λ.
📁 Save: sweep_results.csv + 24 model checkpoints → Drive

---

WEEK 6 — Pareto Surface and Primary Figures

Notebook: notebooks/06_figures.ipynb
Runtime: ~1 hour (CPU fine)

Cell 1 — Load and aggregate

import pandas as pd
df = pd.read_csv(f'{RESULTS_DIR}/sweep_results.csv')
# Compute mean ± 95% CI per λ (bootstrap, 1000 resamples)

Cell 2 — Figure 1: Pareto frontier

# x = efficiency proxy, y = reconstruction MSE
# Each λ = one labeled point + error bars
# Pareto-dominant points connected by dashed line
# Export: fig1_pareto_frontier.pdf at 600 DPI

Cell 3 — Figure 2: Gabor R² vs λ

# x = log(λ), y = Gabor R² mean ± CI
# Horizontal reference line at R² = 0.5
# Export: fig2_gabor_r2_curve.pdf at 600 DPI

Cell 4 — Figure 3: Four-panel joint figure

# (a) MSE vs λ,  (b) sparsity% vs λ
# (c) geometric spread vs λ,  (d) efficiency proxy vs λ
# All panels share the same log-λ x-axis
# Export: fig3_four_panel.pdf at 600 DPI

Cell 5 — Write captions (in notebook text cell)

Write 3 sentences per figure. 
Sentence 1: what is shown. 
Sentence 2: the main result with numbers. 
Sentence 3: interpretation.

✅ Week 6 gate: Three PDF figures exported. At least one figure shows a non-monotonic pattern OR you document a clean null result (both are publishable — see paper/manuscript.docx).
📁 Save: All three PDFs → Drive + figures/ folder in repo

---

WEEK 7 — PCA Comparison and Control Condition

Notebook: notebooks/07_pca_baseline.ipynb
Runtime: ~30 min (CPU fine)

Cell 1 — Fit PCA

from sklearn.decomposition import PCA
pca = PCA(n_components=256).fit(train_patches)  # 80K patches

Cell 2 — PCA reconstruction at k = {64, 128, 256}

for k in [64, 128, 256]:
    # Compute MSE on held-out 20K patches
    # Activation cost = k (always; PCA has no sparsity)

Cell 3 — Apply Gabor-fit to first 16 PCA components

# Expected result: most PCA components R² < 0.3
# If R² > 0.6: your whitening may be incomplete — see Troubleshooting

Cell 4 — Update Figure 1 with PCA points

# Add gray triangle markers for PCA k={64,128,256}
# Export: fig1_pareto_frontier_v2.pdf

Cell 5 — Write comparison note (3 sentences)

Sentence 1: What PCA controls for.
Sentence 2: What the Pareto comparison shows.
Sentence 3: What confound PCA does NOT rule out.

✅ Week 7 gate: Updated Figure 1 with PCA points. 07_pca_gabor_r2.txt saved.

---

WEEK 8 — Literature Review (No Code)

No notebooks. Use Google Scholar, Semantic Scholar, arXiv.

Search strings to use:

"sparse coding Gabor filter quantitative"
"sparse autoencoder V1 simple cell metric"
"natural image statistics sparsity sweep"
"efficient coding Pareto frontier"
"dictionary learning Gabor fit R-squared"

Key papers to read (minimum):

• Olshausen & Field (1996, 1997)
• Bell & Sejnowski (1997)
• Simoncelli & Olshausen (2001)
• Rehn & Sommer (2007)
• Attwell & Laughlin (2001)
• Any SAE papers from 2022–2024 (interpretability literature)

Deliverable: 08_literature_notes.md + 08_gap_statement.txt

---

WEEKS 9–11 — Writing (see paper/manuscript.docx)

Open paper/manuscript.docx. It contains:

• Pre-formatted JCNS template with all section headings
• Numbered placeholders [VALUE_X] for every number from sweep_results.csv
• A fill-in guide at the end of each section

Fill in values in this order:

1. Methods (Week 9) — copy exact values from your code; no interpretation
2. Results (Week 9) — copy mean ± 95% CI from sweep_results.csv; no hedge words
3. Discussion (Week 10) — interpret only; every paragraph must cite a specific number
4. Introduction (Week 10) — write last; must match actual results, not hoped-for results
5. Abstract (Week 10) — write after all other sections; 150–250 words exactly

---

WEEK 12 — Submission

1. Format manuscript using paper/manuscript.docx (already JCNS-formatted)
2. Export all figures as PDF at 600 DPI, confirm legible at 3.5-inch width
3. Write cover letter (template in paper/manuscript.docx Appendix)
4. Submit preprint to bioRxiv: https://www.biorxiv.org/submit
5. Submit to PLOS Computational Biology (primary target, fee waiver available)
6. Record: manuscript ID in 12_submission_log.txt

---

Expected Outputs Per Week

Week	Key Output File	Verification Test
1	patches_zca.npy	Covariance diagonal ≈ 1.0 ± 0.05
2	02_model_lambda001.pt	Dead units < 50/512
3	gabor_utils.py	Synthetic R² > 0.95; noise R² < 0.3
4	metrics_utils.py	Both ordering tests pass
5	sweep_results.csv (24 rows)	Efficiency varies across λ
6	Three figure PDFs	≥1 non-monotonic trend OR documented null
7	Updated Fig 1 + PCA metrics	PCA PCs have lower Gabor R² than SAE atoms
8	08_gap_statement.txt	Gap is specific, not vague
9	09_methods_results_draft.docx	Zero placeholder values remaining
10	10_full_draft_v1.docx	Abstract ≤ 200 words with ≥ 2 numbers
11	11_self_review_checklist.txt	≥ 12 issues found and fixed
12	Preprint URL + MS ID	Confirmation email received

---

Troubleshooting

Week 1: Whitening assert fails

• Most likely cause: np.cov called without transpose — it expects (D, N), not (N, D)
• Second cause: patches not zero-meaned before whitening
• Fix order: (1) check transpose, (2) add mean subtraction, (3) re-run

Week 2: Dead units > 150

• Fix 1: Reduce λ from 0.01 to 0.001
• Fix 2: Add weight normalization after each optimizer step
• Fix 3: Switch from tied to separate encoder/decoder weights

Week 2: Dictionary is noise/uniform blobs

• Confirm whitening passed (Week 1 assert)
• Confirm learning rate is 1e-3, not 1e-2 or 1e-4

Week 3: curve_fit fails on >50% of filters

• Add maxfev=5000 to curve_fit call
• Tighten σ bounds: σ ∈ [0.5, 6]
• Increase multi-start initializations from 5 to 10

Week 5: CSV has fewer than 24 rows after 2 sessions

• The skip logic resumes from where it stopped
• Re-run Cell 2 in a new session — it will only train missing (λ, seed) pairs

Week 5: Efficiency proxy identical for all λ

• The metrics_utils.py functions are not being called correctly
• Add print(f"Computed efficiency: {eff}") after each metric call
• Confirm mse(λ=0) baseline was computed on the same held-out patch set

---

Citation

If you use this codebase, please cite the associated paper (DOI to be added after publication):

@article{YourName2026sparse,
  title   = {Quantifying the Efficiency-Geometry-Similarity Surface in 
             Sparse Autoencoders Trained on Natural Images},
  author  = {[Your Name]},
  journal = {Journal of Computational Neuroscience},
  year    = {2026},
  doi     = {TO BE ADDED}
}

---

Questions? Open a GitHub Issue. The greatest risk is not that the experiment fails — it is that the measurement instruments are wrong and you do not discover it until a reviewer tells you.