Implementing Fine-Tuning and Prompt-Tuning for TabPFN

[tleemann]

Creating a pull request for Finetuning and Prompt tuning.

What has not been done yet, but might be nice to have:

Create tests, check typing with mypy

[CLAassistant]

All committers have signed the CLA.

[iivalchev]

@tleemann Kudos for making the effort! Please note some work for fine-tunning is being done here: https://github.com/LennartPurucker/finetune_tabpfn_v2, not sure if your approach is somewhat different but the change looks quite large. In the repo referenced the goal is not to interfere with the main code base for the purposes of fine-tuning.

[tleemann]

@tleemann Kudos for making the effort! Please note some work for fine-tunning is being done here: https://github.com/LennartPurucker/finetune_tabpfn_v2, not sure if your approach is somewhat different but the change looks quite large. In the repo referenced the goal is not to interfere with the main code base for the purposes of fine-tuning.

@iivalchev Thanks for pointing to your codebase! Don't worry, I was in touch with the maintainers before creating this request.
We thought it could be nice to additionally have native fine-tuning support with the changes in this request, which directly allows for:

• Finetuning on multiple datasets
• Efficient parallel preprocessing of multiple datasets in native torch.DataLoaders
• Prompt-Tuning

[iivalchev]

Right! I am quite keen on the strategies for training on multiple datasets. Is there anything done in that regard?

[tleemann]

Yes, that's a challenging task. The idea here was to preprocess the datasets offline for additional speed through parallelism (preprocessing uses numpy). The classifier now has a get_preprocessed_datasets-function, which returns a Meta-Dataset that can be used in a torch.utils.data.DataLoader to quickly load different preprocessed datasets (or batches of a single dataset) for finetuning.

Please have a look at the example in examples/tabpfn_finetune.py (in the request), where I tried to give an example of the usage (the example only uses one dataset, but it can be easily extended by just passing a list of datasets the get_preprocessed_datasets-function). Hope this helps.

Best,
Tobias

[iivalchev]

Cool! Will do so. Do you think test-time training can also be accommodated? And I assume fine-tunning for the regressor would also be supported?