added an adaptive batching mechanism to handle large test sets based on estimated memory usage

[Krishnadubey1008]

This PR fixes #125

Decription

added an adaptive batching mechanism to handle large test sets based on estimated memory usage. The default behavior can be overridden by expert users by adjusting the memory_saving_mode parameter.

changes made

1. Added a method to estimate memory usage.
2. Modified the predict_proba method to use adaptive batching.

[Copilot]

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[Copilot]

The slice expression 'X[i:i + batch_size]' is not assigned or used, which likely means the intended batch processing is incomplete. Consider assigning the slice to a variable (e.g. batch = X[i:i + batch_size]) and then processing it through the executor.

Suggested change

	X[i:i + batch_size]
	for output, config in self.executor_.iter_outputs(
	X,
	batch = X[i:i + batch_size]
	for output, config in self.executor_.iter_outputs(
	batch,

[noahho]

Hi @Krishnadubey1008,

Thanks for tackling issue #125 and adding the adaptive batching mechanism.

To get this merged, could you please address the following points:

Memory Estimation Function: Please move the memory estimation logic into a reusable function within our utils repository. This helps keep common utilities centralized. You can use this implementation as a reference: https://github.com/PriorLabs/tabpfn_common_utils/blob/524cee72cc6f33cf59fc943dc3e4b5428f3a79bc/expense_estimation.py#L9
CI Checks: The automated tests and the Ruff linter are currently failing. Please investigate the errors shown in the CI checks logs and apply the necessary fixes.
Copilot Suggestion: Please review the comment/suggestion made by GitHub Copilot on this PR, as it may contain relevant points.
Let me know if you have any questions!