Case Study: Criteo dataset

[mrocklin]

The Criteo dataset is a 1TB dump of features around advertisements and whether or not someone clicked on the ad. It has a both dense and categorical/sparse data. I believe that the data is freely available on Azure.

There are some things that we might want to do with this dataset that are representative of other problems:

1. Logistic regression on large sparse data. This could use existing algorithms like L-BGFS or ADMM or it could use the more recent Incremental SGD work. It would be useful to compare the effectiveness of the algorithms above
2. We could also add hyper parameter optimization
3. Gradient boosted trees, presumably with the dask-xgboost connection. This raises a couple of questions. Can XGBoost support categorical data or scipy.sparse arrays? Or perhaps we have to provide a column of integers

As always, it might be a good start to just download a little bit of the criteo dataset (I think that each day of data is available separately) and work with sklearn directly to establish a baseline.

This came out of conversation with @ogrisel

[stsievert]

The dataset is available on Azure (source). This dataset has 13 integer features with 26 categorical features (the categorical features are hashed for anonymization) and is stored as 24 individual files (one for each day) (source).

I would rather use some SGD implementation rather than L-BGFS or ADMM because IIRC those algorithm use the entire dataset in every optimization step.

[mrocklin]

I suspect that for initial use we'll want to play with a single day. I played with this once and it looks like I kept around a conversion script to parquet:

import dask.dataframe as dd
from dask.distributed import Client
client = Client()

categories = ['category_%d' % i for i in range(26)]
columns = ['click'] + ['numeric_%d' % i for i in range(13)] + categories

df = dd.read_csv('day_0', sep='\t', names=columns, header=None) 

encoding = {c: 'bytes' for c in categories}
fixed = {c: 8 for c in categories}
df.to_parquet('day-0-bytes.parquet', object_encoding=encoding,
              fixed_text=fixed, compression='SNAPPY')

[mrocklin]

Although even a single day was a bit of a pain to work with on a single machine if memory serves. It'll be fun to eventually scale up when we're ready.

Presumably we'll start with whatever seems easiest to get decent results from, (as you say SGD) but will eventually want to try a few things. I imagine that some of the preprocessing steps will be the same.

I also recall that when I tried things I got really poor scores. Someone told me this was because clicks were rare and I wasn't weighting them properly.

I suspect that this will force a few topics:

1. Preprocessing
2. Conversion to sparse matrices (Test that dask collections can hold scipy.sparse arrays dask#3738 might help)
3. Some incremental learning with SGD as you suggest
4. Maybe early stopping?
5. Maybe hyper-parameter optimization?

This is also the part of the process (building a pipeline and model) about which I know very little. I'm looking forward to seeing what you end up doing. My guess is that it will be easier to iterate with a small amount of data, probably by just picking off a small bit of the parquet data with pandas and using scikit-learn natively.

[stsievert]

And they're on Kaggle too: https://www.kaggle.com/c/criteo-display-ad-challenge. This will allow good evaluation of our model, with a predefined score function, cross entropy loss. Top scores on Kaggle are around 0.44 (lower is better).

probably by just picking off a small bit of the parquet data with pandas and using scikit-learn natively.

That's my plan for today. I'll try to develop a preprocessing pipeline and reasonable model and test it an a subset of the data.

[stsievert]

Here’s a gist of a simple pipeline I put together: https://gist.github.com/stsievert/30702575de95328f199ab1d7e50795ef

Notes:

• this dataset is unbalanced, with only 3% positive examples.
• I take a (very) small fraction of the dataset, I think about 0.002%.
• I do some preprocessing to avoid outliers. I should probably make it less complex.
• The scores are very variable. I suspect we need more data.

[TomAugspurger]

Thanks for this. I'm playing around with it locally now.

Thinking about scaling this to larger datasets:

1. dd.get_dummies doesn't support sparse. Opened SparseDataFrame empty slice coerces to loses dtype and fill_value pandas-dev/pandas#21993 for that, will fix it later today (have to work around a pandas bug)
2. Did you look at dask_ml.preprocessing.Categorizer and dask_ml.preprocessing.DummyEncoder? With this sparse of data, I think it'll be important to use (pandas) categoricals so that the transformer doing the one-hot / dummy encoding has a consistent shape on train / test sets.
3. I wonder if any of http://contrib.scikit-learn.org/imbalanced-learn/stable/api.html are easily scalable to larger datasets (cc. @glemaitre if he has any quick thoughts here)
4. You might want to have your LogisticProbs.score return the negative log loss. I think as currently written, if you tried to do hyperparameter optimization it would go the wrong direction.

I'll look into the sparse dask get_dummies now, and check back later.

[TomAugspurger]

I do some preprocessing to avoid outliers. I should probably make it less complex.

We have QuantileTransformer. Does that suffice? Hmm, perhaps not, as that only works with dask arrays of known shape, which we may not have at that point...

[glemaitre]

wonder if any of http://contrib.scikit-learn.org/imbalanced-learn/stable/api.html are easily scalable to larger datasets (cc. @glemaitre if he has any quick thoughts here)

If you have a nearest neighbors algorithm which is distributed then it should be possible to make a distributed SMOTE. The under-sampling also rely on NN so it would be most probably possible to scale.

[TomAugspurger]

Thanks. We don't yet have a nearest neighbors algorithm yet, but it's a TODO (also blocking HDBSCAN).

---

@stsievert regarding pd.get_dummies, you may want to just avoid sparse=True. Turns out it's slow (pandas-dev/pandas#21997), and I'm not entirely sure how much memory it's actually saving... profiling now.

[mrocklin]

@TomAugspurger is there another approach that would you suggest? Constructing a scipy.sparse matrix manually? Or are there Scikit-Learn transformers to help with this?

…

On Fri, Jul 20, 2018 at 11:55 AM, Tom Augspurger ***@***.***> wrote: Thanks. We don't yet have a nearest neighbors algorithm yet, but it's a TODO (also blocking HDBSCAN). ------------------------------ @stsievert <https://github.com/stsievert> regarding pd.get_dummies, you may want to just avoid sparse=True. Turns out it's slow ( pandas-dev/pandas#21997 <pandas-dev/pandas#21997>), and I'm not entirely sure how much memory it's actually saving... profiling now. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#295 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszHF3RwBM-YNdRNBme7F8usWU_Q3Gks5uIf1fgaJpZM4VQUPu> .

[TomAugspurger]

For the case study, I recommend we go down all paths simultaneously, to find things that are broken :)

1. Try with dd.get_dummies(sparse=False) and just pay the memory cost. You'll convert to dense anyway, since we have a mixture of sparse and dense (though you do a bit a filtering before that point, so this will have higher memory usage I think).
2. Implement dd.get_dummies(sparse=True) (coming soon) and see where things go
3. Try OrdinalEncoder on the categorical features (should work today with scikit-learn & dask-ml), though this likely won't perform as well with a linear model
4. Use scikit-learn dev & the new OneHotEncoder (dask version coming later today)

[mrocklin]

If memory serves there are hundreds of thousands of categories. Dense may be tricky?

…

On Fri, Jul 20, 2018 at 12:25 PM, Tom Augspurger ***@***.***> wrote: For the case study, I recommend we go down all paths simultaneously, to find things that are broken :) 1. Try with dd.get_dummies(sparse=False) and just pay the memory cost. You'll convert to dense anyway, since we have a mixture of sparse and dense (though you do a bit a filtering before that point, so this will have higher memory usage I think). 2. Implement dd.get_dummies(sparse=True) (coming soon) and see where things go 3. Try OrdinalEncoder on the categorical features (should work today with scikit-learn & dask-ml), though this likely won't perform as well with a linear model 4. Use scikit-learn dev & the new OneHotEncoder (dask version coming later today) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#295 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszETizEUr-EASxO-W1_0qzW2qELE0ks5uIgR-gaJpZM4VQUPu> .

[TomAugspurger]

Ah, there may be many not observed in the sample. In that case, we'll push down the sparse side of things.

[jakirkham]

Regarding nearest neighbors, there is some brute force stuff in dask-distance, which might be useful. It probably could benefit from tree approaches as well if people are interested in that problem.