We are working with a large dataset that fits only once into memory. We have (in memory) a np.ndarray of ~ 90mio x 3k float32s, a list of feature_names, and a list of tuples (categorical_name, num_categories) of features that are categorical, with possible categories 0, ..., num_categories - 1. No missings. Our current code looks like this:
data: np.ndarray
model = GeneralizedLinearRegressor(...)
# As `data` is one block, this should not do any BlockManager shenanigans.
data = pd.DataFrame(data, columns=feature_names, copy=False)
# This might trigger some BlockManager shenanigans
data[categorical_names] = data[categorical_names].astype("category")
# This needs to be done as the train/test data is not an iid split, so we expect
# many unseen categories.
for (category_name, num_categories) in [("a", 2), ....]:
data[category_name] = data[category_name].cat.set_categories(list(range(n_categories)))
model.fit(data, y)Is there an alternative going directly from the np.ndarray to a tabmat.SplitMatrix without any copies (for the non-categorical part)? Pseudo-code:
data_continuous = tabmat.from_csc(data[:, continuous_mask], column_names=...)
data_categorical = ???
data = tabmat.concat([data_continuous, data_categorical])
We are working with a large dataset that fits only once into memory. We have (in memory) a
np.ndarrayof ~ 90mio x 3kfloat32s, a list offeature_names, and a list of tuples(categorical_name, num_categories)of features that are categorical, with possible categories0, ..., num_categories - 1. No missings. Our current code looks like this:Is there an alternative going directly from the
np.ndarrayto atabmat.SplitMatrixwithout any copies (for the non-categorical part)? Pseudo-code: