[New Model] TimesFM-2.5 Foundation Model

[daidahao]

A sub issue of #2933.

Is your feature request related to a current problem? Please describe.
In September, Google released TimesFM 2.5, a 200M-parameter pre-trained time series model, on GitHub and HuggingFace.

The model itself supports univariate series only. Google has recently added covariate support by using xreg (a wrapper for ridge regression at inference time).

TimesFM2.5 currently ranks second on GIFT-EVAL among single open-source models, after Chronos-2 (implemented in #2943 #2961). If implementing a TimesFM 2.5 is of interest of Darts, I am happy to contribute a PR.

Describe proposed solution
Porting TimesFM-2.5 to FoundationModel should be easy and requires no additional dependencies than those already included (huggingface_hub for downloading weights, safetensors for loading weights, and torch). However, supporting covariates could be a challenge:

In Google's repo, there are two modes for adding covariate support, aka, xreg_mode:

• "xreg + timesfm" (default): fits a model on the residuals of the TimesFM forecast
• "timesfm + xreg": fits a model on the targets then forecasts on the residuals via TimesFM

Either mode involves solving ridge regression on targets and covariates at inference time (see forecast_with_covariates() in timesfm_2p5_base.py). We might not be able to do so in Darts because this might cause data leakage in the same batch (e.g., when calling historical_forecasts()).

Instead, I suggest we use a linear head and a MSE loss with L2 regularization to train that head and mimic ridge regression.

Another potential challenge is of course fine-tuning #2964. Since fine-tuning would require a deterministic loss or a Quantile Regression, this may not be compatible with the MSE loss. I haven't thought a way to address the conflict. Perhaps we get rid of the MSE loss when fine-tuning is enabled but this would make the model behave differently than not fine-tuning.

Describe potential alternatives
A clear and concise description of any alternative solutions or existing features that might solve it.

Additional context
Overall, a few discussions points I can think of:

• Should we support covariates for TimesFM-2.5?
• If so, should we add a linear head and a MSE loss with L2 to train the head? Are there better ways of doing this without risking data leakage?
• What modes would the model support ("xreg + timesfm", "timesfm + xreg", or both) and which is the default?
• Should the model support fine-tuning? If so, would it be compatible with the MSE loss?

I've tried to reach out to Google about collaborations but they haven't responded. See google-research/timesfm#348. I will invite TimesFM-2.5 co-authors for discussions here.

[daidahao]

@rajatsen91 @siriuz42
Thank you for your work on TimesFM!

Darts is a popular time series library with many capabilities such as forecasting and classification.
I've working on implementing state-of-the-art foundation models in Darts so that 1) it may serve as a neutral ground for foundation models (envisioned by @hrzn) and 2) users can apply pre-trained and trained forecasting models interchangeably under a unified API.

I would like to invite you for discussions on the issues here. Particularly, I am interested in your opinions on using TimesFM-2.5 with covariates. If we were to choose one xreg_mode for regression, which one would you recommend? Also, what would be the best practice for fine-tuning TimesFM-2.5 with covariates?

[brunnedu]

Hi @daidahao,

Thanks for taking the initiative to add the TimesFM 2.5 foundation model to Darts!

@dennisbader is currently on leave, but I’d like to share some initial thoughts to keep the discussion moving. One significant point of discussion is the limitation to univariate time series and the fact that covariate support is implemented through a "hybrid" ridge regression rather than being native to the transformer architecture. As Dennis mentioned in the original thread about foundation model support (#2933):

For other models, I would suggest prioritising models with multivariate and co-variate support since they are more likely to benefit Darts users.

---

That said, here are my thoughts on the specific questions you raised (though I'm looking forward to @dennisbader's opinion as well):

Should we support covariates for TimesFM-2.5?

Ideally, yes. Supporting covariates would make the model more useful for Darts users.

If so, should we add a linear head and a MSE loss with L2 to train the head? Are there better ways of doing this without risking data leakage?

I appreciate your caution about data leakage which indeed seems to be an important concern here. However, adding a linear head effectively turns this into a supervised task requiring fine-tuning for every new set of covariates. If we want to preserve the zero-shot/foundation nature of TimesFM, we should implement the Ridge Regression (OLS) solution as an 'inference-time' adapter. To avoid the data leakage you mentioned, we can ensure the regression parameters (β) are estimated strictly on the context window of each sample in the batch.

What modes would the model support ("xreg + timesfm", "timesfm + xreg", or both) and which is the default?

Supporting both modes is preferable as they handle the target-covariate relationship differently. However, I think we can first prioritize the default "xreg + timesfm" mode and potentially add the other mode later if there's demand.

Should the model support fine-tuning? If so, would it be compatible with the MSE loss?

Ideally, all foundation models would support fine-tuning, but for now I think we can focus on getting the basic implementation right first and tackle fine-tuning as a future enhancement.

---

I have a few questions to better understand the proposed implementation:

• Is my understanding correct that when users want to use covariates, they would first need to call .fit() to learn the ridge regression weights? This could be in conflict with the intuitive FoundationModel API of not requiring .fit() before calling .predict(). How do you envision handling this?

• Performance vs. Chronos-2: While TimesFM-2.5 ranks second on GIFT-EVAL, have you had a chance to compare its performance characteristics (speed, memory usage, accuracy) against Chronos-2 on some of our Darts evaluation datasets? This would provide some interesting insights in how many users would potentially use TimesFM over Chronos-2.

Thanks again for your efforts, looking forward to hearing your thoughts!

[rajatsen91]

Thanks for looking into this. I agree with most of the TimesFM-2.5 related implementation suggestion made by @brunnedu.

In particular for zero-shot covariate:

1. On an average we see the best performance from xreg+timesfm mode that is implemented here.

For fine-tuning without covariates:

2. We can just do encoder-only fine-tuning fairly easily. We have not released example code for this for TimesFM-2.5 but for the 2.0 model this notebook was an example.

For fine-tuning with covariates:

3. Some good choices would be: (i) convert all covariates available in the past + past of the time-series in contiguous patches of input_patch_length (=32) into a token for the main transformer using a MLP encoder. This can be done for every patch. (ii) The decoder can map the last output token to the future + there could be several choices for taking into account the covariates available in the future.

[dennisbader]

Thanks @daidahao, @brunnedu and @rajatsen91 for driving this discussion.

We indeed favor neural network models with covariate support. In my opinion, we should not add the "regression" part / covariate support to the model itself, since it would go against the torch-only approach the we currently follow. Adding support for this would mean that we need to refactor some of the train and predict logic, which I'm not convinced by.

Here is an example of how the user could achieve xreg+timesfm (note that here I use Chronos2 instead of TimesFM):

python code

import matplotlib.pyplot as plt
from sklearn.linear_model import Ridge

import darts.utils.timeseries_generation as tg
from darts.datasets import AirPassengersDataset
from darts.models import SKLearnModel, Chronos2Model

context_length = 36
horizon = 12

# target series
series = AirPassengersDataset().load().astype("float32")

# example of explicit covariates
covs = tg.linear_timeseries(
    start=series.start_time(),
    length=len(series),
    freq=series.freq
).astype("float32")

# train and val split (relevant covariate time frames will be extracted automatically by models)
series_train, series_val = series[:-horizon], series[-horizon:]

# create the regression model
reg_model = SKLearnModel(
    model = Ridge(),
    lags_future_covariates=[0],  # only use future covariates, not target itself
    add_encoders={"cyclic": {"future": ["month"]}},  # add automatic covariates
)

# fit reg model on automatic and explicit covariates
reg_model.fit(
    series=series_train,
    future_covariates=covs
)

# predict all historic target values using only future covariates (insample prediction)
series_reg = reg_model.historical_forecasts(
    forecast_horizon=1,
    series=series_train,
    future_covariates=covs,
    retrain=False,
)
# compute residuals
series_residuals = series_train - series_reg

# forecast using the regression model
forecast_reg = reg_model.predict(
    n=horizon,
    series=series_train,
    future_covariates=covs,
)

# create a foundation model
foundation_model = Chronos2Model(input_chunk_length=context_length, output_chunk_length=horizon)

# fit (only used to load the model) and forecast the residuals
forecast_residuals = foundation_model.fit(series_residuals).predict(
    n=horizon,
    series=series_residuals,
)

# add back regression forecasts
forecast = forecast_residuals + forecast_reg

# plot results
series.plot(label="actual")
forecast.plot(label="forecast (xreg+chronos)")
plt.show()

Of course there is a bit of boilerplate code, but in general this gives complete flexibility on how to generate the residuals before feeding it to the foundation model (e.g.: which regressor to use? Can be any of our SKLearnModels (LGBM, CatBoost, LinearRegression, ...)

To @daidahao's questions:

Should we support covariates for TimesFM-2.5?

I would suggest for now only to implement the model without covariate support.

If so, should we add a linear head and a MSE loss with L2 to train the head? Are there better ways of doing this without risking data leakage?

The approach I showed above would not have any data leakage, since it only uses future covariates at time t to predict the target at time t.

Should the model support fine-tuning? If so, would it be compatible with the MSE loss?

For now, it does not have to support fine-tuning. I would be great to find a future-proof solution for all models in #2964.