A small, self contained study of self supervised learning. The question is simple: does a representation learned without labels actually carry useful structure, or could you get the same thing from a randomly initialized network? This repo answers that with a tiny SimCLR encoder and a linear probe, and it runs in a few seconds on a CPU with no downloads.
Self supervised learning trains an encoder using the data alone, with no labels. SimCLR does this by contrastive learning. You take a batch of images, make two randomly augmented views of each one, and ask the encoder to pull the two views of the same image together in feature space while pushing different images apart. The loss that expresses this is NT-Xent, a temperature scaled cross entropy over cosine similarities.
Once the encoder is pretrained you freeze it and check how good its features are with a linear probe. A linear probe is just a logistic regression fit on the frozen features of a small labeled set. If the features are good, a linear model on top of them already separates the classes. The honest baseline is a probe on the features of the very same architecture left untrained, so any gap you see comes from the pretraining and not from the network shape.
Everything is synthetic so the whole thing stays fast and offline. Each class is a fixed random prototype image. A sample is its prototype plus noise, and the augmentation during pretraining adds more noise together with random brightness and contrast shifts. With the noise turned up the classes overlap enough that a random encoder cannot separate them cleanly, which is exactly the regime where pretraining should help.
src/
data.py prototypes, dataset construction, augmentation
model.py TinyEncoder, ProjectionHead, SimCLRModel
simclr.py NT-Xent loss and the pretraining loop
probe.py feature extraction and the logistic regression probe
experiment.py end to end run: random baseline versus SimCLR
tests/
test_loss.py properties of the NT-Xent loss
test_pretraining.py loss decreases over steps
test_probe.py SimCLR probe beats the random probe, and is reproducible
Run the full experiment and print the numbers:
python -m src.experiment
Run the tests:
python -m pytest tests/ -q
These are the numbers from running python -m src.experiment on CPU at the
default settings (seed 0). Your exact figures will match because the run is
seeded.
pretraining loss start (mean first 5): 4.8405
pretraining loss end (mean last 5): 3.8965
linear probe accuracy, random encoder: 0.7266
linear probe accuracy, SimCLR encoder: 0.9609
The pretraining loss falls steadily, and the probe on the SimCLR features clears the random encoder baseline by a wide margin. Checking seeds 0, 1, and 2 the SimCLR encoder wins every time, so the result is not an artifact of one lucky initialization.
The tests are behavioral rather than checks against frozen magic numbers.
- The NT-Xent loss is finite and positive, has a gradient, and gives a lower value when the two views are aligned than when they are unrelated.
- Pretraining reduces the loss: the average over the last steps is well below the average over the first steps.
- The linear probe on SimCLR features beats the probe on random features, and the whole pipeline is reproducible across two calls with the same seed.
This is a teaching scale setup. The encoder is a two block CNN and the data is synthetic, so the absolute accuracies say nothing about real images. The point is the comparison, which holds: learning a representation without labels beats starting from a random one. Scaling up would mean a real dataset, a larger backbone, more augmentations, and a longer schedule, but the moving parts stay the same.