OpenLTM (Open-Source Large Time-Series Models)

Large time-series models, pre-training datasets, and adaptation techniques.

Note

OpenLTM is a open codebase intending to explore the model architecture of large time-series models. It is not intended to be completely compatiable with official codebases and existing checkpoints. We aim to provide a neat pipeline to develop and evaluate large time-series models, which covers three milestone applications: supervised training, large-scale pre-training, and model adaptation.

For deep time series models and task-specific benchmarks, we strongly recommend Time-Series-Library and this comprehensive Survey.

🚩 News (2024.10) We include four large time-series models, release pre-training logic, and provide scripts.

What is LTM

LTM (Large Time-Series Model) is a series of scalable deep models built on foundation backbones (e.g. Transformers) and large-scale pre-training, which will be applied to a variety of time series data and diverse downstream tasks. For more information, here we provide [Slides]!

Model Checklist

• Moirai - Unified Training of Universal Time Series Forecasting Transformers. [ICML 2024], [Code]
• Moment - MOMENT: A Family of Open Time-series Foundation Model. [ICML 2024], [Code]
• Timer - Timer: Generative Pre-trained Transformers Are Large Time Series Models. [ICML 2024], [Code]
• Timer-XL - Timer-XL: Long-Context Transformer for Unified Time Series Forecasting. [arxiv 2024], [Code]

We will update the following models to the checklist after a comprehensive evaluation. Welcome to give your suggestion about any interesting works 🤗

• AutoTimes: Autoregressive Time Series Forecasters via Large Language Models. [NeurIPS 2024], [Code]
• Chronos: Learning the Language of Time Series. [arxiv 2024], [Code]
• Time-MoE: Billion-Scale Time Series Foundation Models With Mixture Of Experts. [arxiv 2024], [Code]
• A Decoder-Only Foundation Model for Time-Series Forecasting. [arxiv], [Code]

Usage

1. Install Python 3.10. For convenience, execute the following command.

pip install -r requirements.txt

1. Place downloaded data in the folder ./dataset. Here is a dataset summary.

• For pre-training:

  • UTSD contains 1 billiion time points for large-scale pre-training (in numpy format): [Download].
  • ERA5-Familiy (40-year span, thousands of variables) for domain-specific model: [Download].

• For superwised training or modeling adaptation

  • Well-acknowlegded datasets from TSLib : [Download].

2. We provide pre-training and adaptation scripts under the folder ./scripts/. You can conduct experiments using the following examples:

# Supervised training
# (a) one-for-one forecasting
bash ./scripts/supervised/forecast/moirai_ecl.sh
# (b) one-for-all (rolling) forecasting
bash ./scripts/supervised/rolling_forecast/timer_xl_ecl.sh

# Large-scale pre-training
# (a) pre-training on UTSD
bash ./scripts/pretrain/timer_xl_utsd.sh
# (b) pre-training on ERA5
bash ./scripts/pretrain/timer_xl_era5.sh

# Model adaptation
# (a) full-shot fine-tune
bash ./scripts/adaptation/full_shot/timer_xl_etth1.sh
# (b) few-shot fine-tune
bash ./scripts/adaptation/few_shot/timer_xl_etth1.sh
# (c) zero-shot generalization
bash ./scripts/adaptation/zero_shot/timer_xl_etth1.sh

3. Develop your large time-series model.

• Add the model file to the folder ./models. You can follow the ./models/timer_xl.py.
• Include the newly added model in the Exp_Basic.model_dict of ./exp/exp_basic.py.
• Create the corresponding scripts under the folder ./scripts.

Efficiency

We present a theoretical proof of the computational complexity of Time-Series Transformers. See the paper for details.

Note

LTMs are still small in scale compared to large models of other modalities. We prefer to include and implement models requiring affordable training resources as efficiently as possible (for example, using several RTX 4090s or A100s).

Citation

If you find this repo helpful, please cite our paper.

@inproceedings{liutimer,
  title={Timer: Generative Pre-trained Transformers Are Large Time Series Models},
  author={Liu, Yong and Zhang, Haoran and Li, Chenyu and Huang, Xiangdong and Wang, Jianmin and Long, Mingsheng},
  booktitle={Forty-first International Conference on Machine Learning}
}

Acknowledgment

We appreciate the following GitHub repos a lot for their valuable code and efforts:

• Time-Series-Library (https://github.com/thuml/Time-Series-Library)
• Large-Time-Series-Model (https://github.com/thuml/Large-Time-Series-Model)
• AutoTimes (https://github.com/thuml/AutoTimes)

Contributors

If you have any questions or want to use the code, feel free to contact:

• Yong Liu (liuyong21@mails.tsinghua.edu.cn)
• Guo Qin (qinguo24@mails.tsinghua.edu.cn)