TabPFNRegressor preprocessing fails on bigger datasets fix

[Krishnadubey1008]

This PR fixes #169
chnages made- took n_quantiles=min(n_quantiles, 10_000) in TabPFN/src/tabpfn/model/preprocessing.py

[Copilot]

Pull Request Overview

This PR fixes an issue in the TabPFNRegressor preprocessing module where the number of quantiles could become excessively large on bigger datasets. The change ensures that the quantiles for various QuantileTransformer instances are capped at 10,000.

• Updated the "quantile_uni_coarse" and "quantile_norm_coarse" transformers to use n_quantiles as min(max(num_examples // 10, 2), 10_000).
• Updated the "quantile_norm" transformer to use n_quantiles as min(max(num_examples // 5, 2), 10_000).
• Updated the "quantile_uni_fine" and "quantile_norm_fine" transformers to use n_quantiles as min(num_examples, 10_000).

[noahho]

Thanks so much for this change! Would yu be able to add a test for this change, i.e. one that tests if the preprocessing runs on datasets of > 10,000 samples. We can't run the inference step unfortunately as ofc, it would be way too slow. Only way to test the inference on lareg datasets was if we provided a tiny tabpfn checkpoint, a very small model that is randomly initialized but that would be a project in itself.

[Krishnadubey1008]

@noahho I had added test_preprocessing.py please suggest changes if any

[Copilot]

Pull Request Overview

This PR fixes an issue with the TabPFNRegressor preprocessing on larger datasets by capping the number of quantiles to 10,000 in several QuantileTransformer configurations.

• Updated quantile transformer settings in the preprocessing module to avoid excessive quantile calculations.
• Added a test case in tests/test_preprocessing.py to verify functionality on large datasets.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
tests/test_preprocessing.py	Adds a new test to verify preprocessing functionality on large datasets.
src/tabpfn/model/preprocessing.py	Updates the n_quantiles parameter in various QuantileTransformer configurations by capping values to 10,000.

Comments suppressed due to low confidence (2)

tests/test_preprocessing.py:27

• Consider adding assertions to verify the shape and properties of the transformed output to increase test coverage beyond a simple non-null check.

assert result is not None

src/tabpfn/model/preprocessing.py:722

• [nitpick] The 'quantile_uni_coarse' transformer now caps n_quantiles to 10,000, yet the 'quantile_uni' transformer remains uncapped. If this discrepancy is unintentional, consider applying the same cap or adding a clarifying comment.

n_quantiles=min(max(num_examples // 10, 2), 10_000),

[noahho]

Great, this looks really good. There seems to be a tiny ruff issue at this point. Do you know how to resolve?
"ruff check . --fix" with ruff version 0.8.6

[noahho]

Ohh also something that copilot just caught:
The 'quantile_uni_coarse' transformer now caps n_quantiles to 10,000, yet the 'quantile_uni' transformer remains uncapped.

[Krishnadubey1008]

i will fix it now

[Krishnadubey1008]

@noahho I had ran ruff check . --fix but still ruff linting test is failing

[noahho]

The two open ones don't seem to be automatically fixable:
src/tabpfn/regressor.py:723:89: E501 Line too long (89 > 88)
tests/test_preprocessing.py:12:9: NPY002 Replace legacy np.random.rand call with np.random.Generator

An LLM will know how to fix number 2 and by deleting a character in src/tabpfn/regressor.py:723:89 you fix no1

[Krishnadubey1008]

@noahho Please review

[noahho]

Thanks a lot for continuing to work on this. It seems there were a few changes made for the linting that weren't right (such as adding ""). I'll look into the PR and fix those things, if you'd like me to.

[Krishnadubey1008]

Yes ,sure