Remove unused levels from categorical unique_id

[nickto]

Summary

Check if unique_id is categorical and drop unused levels.

Why?

If unique_id is categorical, and some levels were removed, then value_counts() in _grouped_array_from_df() still returns these levels, but with counts of 0.

_grouped_array_from_df(), however, does not expect that some counts could be zero, therefore, it makes sense to drop unused levels from a categorical index.

This situation arises, for example, when we

1. Use a categorical unique_id, e.g., to save memory when a dataset is large.
2. Sub-sample only some of the unique_id, e.g., for faster development, because the data set is large.

Feel free to use your judgment on whether this edge case is common enough to justify an additional validation.

Test

Example code, which fails before the fix:

from datetime import date, timedelta

import numpy as np
import pandas as pd

from statsforecast import StatsForecast
from statsforecast.models import AutoARIMA

T = 100
N_GROUPS = 3
N_SAMPLE = 2

np.random.seed(2023)

# Generate data
df = pd.DataFrame({
    "unique_id": np.repeat([chr(ord("a") + i) for i in range(N_GROUPS)], T),
    "ds": [date(year=2023, month=1, day=1) + timedelta(days=i) for i in range(T)] * N_GROUPS,
    "y": np.random.normal(0, 1, (N_GROUPS * T))
})

# Cast `unique_id` column to category type
df["unique_id"] = df["unique_id"].astype("category")

# Sample only some of the unique IDs
unique_ids = list(set(df["unique_id"]))
sampled_ids = np.random.choice(unique_ids, 2, replace=False)
df = df.loc[df["unique_id"].isin(sampled_ids)]

# Forecast
sf = StatsForecast(
    df=df,
    models=[AutoARIMA()],
    freq="D",
)
sf.forecast(h=5)

Throws the following error, which does not help identifying the cause of the problem:

Error: ValueError: zero-size array to reduction operation minimum which has no identity

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[CLAassistant]

All committers have signed the CLA.

[AzulGarza]

Hey @nickto! This was a problem we had a long time ago. Thank you very much for solving it! 🙌

LGTM

[AzulGarza]

@all-contributors please add @nickto for code

[allcontributors]

@FedericoGarza

I've put up a pull request to add @nickto! 🎉