Data Leakage

[nova-land]

The use of tf.keras.utils.normalize will provide invalid test result by normalising the whole dataset.

An evaluation script is required to verify the accuracy of the model

[kyleskom]

I don't understand what the issue here is?

[chriseling]

I think the worry is the normalize is applied to the whole data set which could potentially overfit the model because the validation data is also normalized. Best, Chris

…

On Sun, Jun 4, 2023 at 8:48 AM Kyle Skompinski ***@***.***> wrote: I don't understand what the issue here is? — Reply to this email directly, view it on GitHub <#249 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVP5WZGNGLQYZS6PVW3IATXJSU35ANCNFSM6AAAAAAYRZ5LRA> . You are receiving this because you are subscribed to this thread.Message ID: <kyleskom/NBA-Machine-Learning-Sports-Betting/issues/249/1575616982@ github.com>

[kyleskom]

Ill take a look when I revisit this next season

[kyleskom]

Hi looking for more info on what the potential fix for this would be. Thank you.

[nova-land]

You will need to separate train and test data when you are using tf.keras.utils.normalize. But normally you should use Scaler in scikit-learn to separate train and test data, fit the train data then transform both train and test data.

[STRATZ-Ken]

I am not sure I agree with @nova-land. The idea of normalize is to set the data for the entire dataset equally. Imagine you have a data set that has values of [3,1,0.50] and you normalize this. It would change to [1, .33, .165]. If your next dataset has a higher value, it would adjust based on the highest data on the column.

There are keras layers you can do which will normalize the data inside the model itself, which would not require this function to be called. Or you can normalize the data when it comes in, setting max values. For example, if a player scores 56 points, and your goal is predict how many points a player is going to score from 0 to 50 (Your force normalizing here), then the max he can score is 50. Just an example.

I am not an expert here, but you have to make sure you have this code in your training set. Then when your ready to predict, you load these values and send the predictions through the normalize function as well.

if not os.path.exists(model_dir + '/scaler.pkl'):
        joblib.dump(min_max_scaler, model_dir + '/scaler.pkl')

[STRATZ-Ken]

Here is information on the normalize layer. You would add this before your first dense layer, this will normalize the incoming data and store its weights inside the model file itself. Then you would not have to make any changes to the data or even call MinMax normalize within the file itself.

https://keras.io/api/layers/normalization_layers/batch_normalization/

Also worth noting, this is for the NN model, not XGBoost.

[Gxent]

but then which would be better xg or nn model?

[STRATZ-Ken]

Better is not a good word at all to use in models. There are a million factors. That question cannot be answered.

[Gxent]

Okay, put another way. What probability would be closest since I made $2,000 in two weeks via XGboost with just a $10 stake. in the end season in May, and I didn’t pay attention to the NN model...

[Gxent]

so I always relied on over and under

[cafeTechne]

Okay, put another way. What probability would be closest since I made $2,000 in two weeks via XGboost with just a $10 stake. in the end season in May, and I didn’t pay attention to the NN model...

How's this working out for you now?

[Gxent]

this year wasn't so good

[cafeTechne]

this year wasn't so good

So you're not seeing 55% win rates with this strategy?

[etonyetang]

Okay, so the user wants to fix data leakage related to batch normalization in their XGBoost model for NBA odds. Wait, XGBoost doesn't use batch normalization, right? That's a neural network thing. Maybe the user is confused or there's another form of data leakage here. Let me check their code again.

Looking at their code, they're splitting the data into train and test inside a loop 100 times using train_test_split with a test size of 0.1. Then they train XGBoost each time. Oh, but wait, they're not using any time-based split. The data includes dates, and they're randomly splitting, which could cause future data to leak into the training set. That's a problem because in time-series data, you shouldn't use future data to predict the past. That's one source of data leakage.