This is the repository for the BioCLIP model and the TreeOfLife-10M dataset. It contains the code used for training and the evaluation of BioCLIP (testing and visualizing embeddings). Additionally, we include a collection of scripts for forming, evaluating, and visualizing the data used for TreeOfLife-10M and the Rare Species benchmark we created alongside it. The BioCLIP website is hosted from the gh-pages branch of this repository.
BioCLIP is a CLIP model trained on a new 10M-image dataset of biological organisms with fine-grained taxonomic labels. BioCLIP outperforms general domain baselines on a wide spread of biology-related tasks, including zero-shot and few-shot classification.
The BioCLIP model is a ViT-B/16 pre-trained with the CLIP objective. Both the ViT and the (small) autoregressive text encoder are available to download on Hugging Face.
The only dependency is the open_clip package.
See the examples/ directory on the Hugging Face model repo for an example implementation.
You can also use the pybioclip package or the BioCLIP demo on Hugging Face.
BioCLIP was trained on TreeOfLife-10M (ToL-10M). The data is a combination of iNat21, BIOSCAN-1M, and data we collected and cleaned from Encyclopedia of Life (EOL). It contains images for more than 450K distinct taxa, as measured by 7-rank Linnaean taxonomy (kingdom through species); this taxonomic string is associated to each image along with its common (or vernacular name) where available.
We cannot re-release the iNat21 or the BIOSCAN-1M datasets; however, we have uploaded our cleaned EOL data to TreeOfLife-10M on Hugging Face.
After downloading iNat21 and BIOSCAN-1M, the three datasets can be combined into TreeOfLife-10M in the webdataset format for model training by following the directions in treeoflife10m.md.
10 biologically-relevant datasets were used for various tests of BioCLIP, they are described (briefly) and linked to below. For more information about the contents of these datasets, see Table 2 and associated sections of our paper. Annotations used alongside the datasets for evaluation are provided in subfolders of the data/ directory named for the associated dataset.
- Meta-Album: Specifically, we used the Plankton, Insects, Insects 2, PlantNet, Fungi, PlantVillage, Medicinal Leaf, and PlantDoc datasets from Set-0 through Set-2 (Set-3 had not yet been released).
- Birds 525: We evaluated on the 2,625 test images provided with the dataset.
- Rare Species: A new dataset we curated for the purpose of testing this model and to contribute to the ML for Conservation community. It consists of nearly 12K images representing 400 species labeled Near Threatened through Extinct in the Wild by the IUCN Red List. For more information, see our Rare Species dataset.
We have a preprint on arXiv and a project website. We also will link to the upcoming CVPR 2024 version when it is publicly available.
The docs/ directory is divided into two subfolders: imageomics/ and open_clip/. The former is documentation relating to the creation of BioCLIP, TreeOfLife-10M, and the Rare Species dataset, while the latter is documentation from the open_clip package (this has not been altered).
We plan on adding more docs on how to use BioCLIP in a variety of settings.
For now, if it is unclear how to integrate BioCLIP into your project, please open an issue with your questions.
Our paper:
@inproceedings{stevens2024bioclip,
title = {{B}io{CLIP}: A Vision Foundation Model for the Tree of Life},
author = {Samuel Stevens and Jiaman Wu and Matthew J Thompson and Elizabeth G Campolongo and Chan Hee Song and David Edward Carlyn and Li Dong and Wasila M Dahdul and Charles Stewart and Tanya Berger-Wolf and Wei-Lun Chao and Yu Su},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2024},
pages = {19412-19424}
}
Our code (this repository):
@software{bioclip2023code,
author = {Samuel Stevens and Jiaman Wu and Matthew J. Thompson and Elizabeth G. Campolongo and Chan Hee Song and David Edward Carlyn},
doi = {10.5281/zenodo.10895871},
title = {BioCLIP},
version = {v1.0.0},
year = {2024}
}
Also consider citing OpenCLIP, iNat21 and BIOSCAN-1M:
@software{ilharco_gabriel_2021_5143773,
author={Ilharco, Gabriel and Wortsman, Mitchell and Wightman, Ross and Gordon, Cade and Carlini, Nicholas and Taori, Rohan and Dave, Achal and Shankar, Vaishaal and Namkoong, Hongseok and Miller, John and Hajishirzi, Hannaneh and Farhadi, Ali and Schmidt, Ludwig},
title={OpenCLIP},
year={2021},
doi={10.5281/zenodo.5143773},
}
@misc{inat2021,
author={Van Horn, Grant and Mac Aodha, Oisin},
title={iNat Challenge 2021 - FGVC8},
publisher={Kaggle},
year={2021},
url={https://kaggle.com/competitions/inaturalist-2021}
}
@inproceedings{gharaee2023step,
author={Gharaee, Z. and Gong, Z. and Pellegrino, N. and Zarubiieva, I. and Haurum, J. B. and Lowe, S. C. and McKeown, J. T. A. and Ho, C. Y. and McLeod, J. and Wei, Y. C. and Agda, J. and Ratnasingham, S. and Steinke, D. and Chang, A. X. and Taylor, G. W. and Fieguth, P.},
title={A Step Towards Worldwide Biodiversity Assessment: The {BIOSCAN-1M} Insect Dataset},
booktitle={Advances in Neural Information Processing Systems ({NeurIPS}) Datasets \& Benchmarks Track},
year={2023},
}