More Flexibility for train_test_validate

[jjacobson95]

In some cases, a user may want to split the data into two groups (train/validate or train/test). Currently train_test_validate requires 3 groups, but it would be great if it could take different numbers and still use the same stratification and split_type logic.

Main example: A user wants to train/validate on a single dataset and then apply to another dataset (Transfer Learning).

[ymahlich]

I might fundamentally misunderstand transfer learning here, but don't you still need a train/test/validate set in transfer learning to do feature optimization?

I certainly can implement another method that does only split two ways, I am just questioning how useful this will be.

[jjacobson95]

I believe 'train' builds the model, 'validate' does feature optimization/tuning, and the 'test' set doesn't impact model performance as the model is frozen at this point.

So for example, we could train/validate on CCLE and then test on NCI60, BeatAML, or whatever dataset we'd like.

• In this example, if we did train/test/validate/ on CCLE, we'd be losing 10% of the data (in test) that could be used to train or validate.