Model description
Summary : Next-Embedding Prediction: The Simple Secret to Strong Vision Learners
NEPA is a self-supervised method. It trains Vision Transformers to predict future patch embeddings. No complex loss functions or extra heads. Achieves 85.3% top-1 accuracy on ImageNet-1K with ViT-L.
The NEPA model got released two days ago and while the results are promising, I couldn't not notice how close to transformers the code they released was, and as @NielsRogge mentioned in his tweet, they heavily relied on the HF libraries.
So I suggest we could add this model to transformers and showcase how easy it becomes easy for researchers to pick and build on top of existing models. Anyway, I got a first draft PR to be opened.
Open source status
Provide useful links for the implementation
Every useful links are in the HF paper page : https://huggingface.co/papers/2512.16922
@yonigozlan @SihanXU
Model description
Summary : Next-Embedding Prediction: The Simple Secret to Strong Vision Learners
NEPA is a self-supervised method. It trains Vision Transformers to predict future patch embeddings. No complex loss functions or extra heads. Achieves 85.3% top-1 accuracy on ImageNet-1K with ViT-L.
The NEPA model got released two days ago and while the results are promising, I couldn't not notice how close to transformers the code they released was, and as @NielsRogge mentioned in his tweet, they heavily relied on the HF libraries.
So I suggest we could add this model to transformers and showcase how easy it becomes easy for researchers to pick and build on top of existing models. Anyway, I got a first draft PR to be opened.
Open source status
Provide useful links for the implementation
Every useful links are in the HF paper page : https://huggingface.co/papers/2512.16922
@yonigozlan @SihanXU