Timer (Large Time Series Model)

This repo provides official code, datasets and checkpoints for Timer: Generative Pre-trained Transformers Are Large Time Series Models. [Poster], [Slides].

Updates

🚩 News (2024.6) Pre-training dataset (UTSD) is available in HuggingFace. Dataloader is also contained.

🚩 News (2024.5) Accepted by ICML 2024, a camera-ready version of 31 pages.

🚩 News (2024.4) The pre-training scale has been extended, enabling zero-shot forecasting.

🚩 News (2024.2) Releasing model checkpoints and code for adaptation.

Introduction

Time Series Transformer (Timer) is a Generative Pre-trained Transformer for general time series analysis. You can visit our Homepage for a more detailed introduction.

Datasets

We curate Unified Time Series Datasets (UTSD) comprised of 1B time points and 4 volumes to facilitate the research on large time series models and pre-training.

Our dataset is released in HuggingFace to facilitate the research of large models and pre-training in the field of time series.

Usage

You can access and load UTSD in the style of TSLib based on the following:

# huggingface-cli login
# export HF_ENDPOINT=https://hf-mirror.com 

python ./scripts/UTSD/download_dataset.py

# dataloader
python ./scripts/UTSD/utsdataset.py

Tasks

Forecasting: We provide all scripts as well as datasets for few-shot forecasting in this repo.

Imputation: We propose segment-level imputation, which is more challenging than point-level imputation.

Anomaly Detection: We provide new benchmarks of predictive anomaly detection on UCR Anomaly Archive.

We provide detailed README files illustrating each task under the folder ./scripts/.

Code for Fine-tuning

1. Install Pytorch and necessary dependencies.

pip install -r requirements.txt

2. Put downstream datasets from Google Drive and Tsinghua Cloud under the folder ./dataset/.

3. Put the checkpoint from Google Drive and Tsinghua Cloud under the folder ./checkpoints/.

4. Train and evaluate the model. We provide the above tasks under the folder ./scripts/.

# forecasting
bash ./scripts/forecast/ECL.sh

# segement-level imputation
bash ./scripts/imputation/ECL.sh

# anomaly detection
bash ./scripts/anomaly_detection/UCR.sh

Train on Custom Dataset

To fine-tune on your time series dataset, you can try out the following steps:

1. The essense is to reload the customized dataloader and load the pre-trained checkpoint (See ./scripts/ folder).
2. CIDatasetBenchmark/CIAutoRegressionDatasetBenchmark in the data_provider folder can train and evaluate models in direct / iterative multi-step mode.

Approach

Pre-training and Adaptation

To pre-train on heterogeneous time series, we propose single-series sequence (S3), reserving series variations with the unified context length. Further, we convert forecasting, imputation, and anomaly detection into a unified generative task.

Model Architecture

Given the limited exploration of the backbone for large time series models, we extensively evaluate candidate backbones and adopt the decoder-only Transformer with autoregressive generation towards LTSMs.

Performance

Timer achieves state-of-the-art performance in each task and we present the pre-training benefit on few-shot scenarios.

Scalability

By increasing the parameters and pre-training scale, Timer achieves notable performance improvement: 0.231 $\to$ 0.138 (−40.3%), surpassing the previous state-of-the-art deep forecasters.

Flexible Sequence Length

The decoder-only architecture provides the flexibility to accommodate time series of different lookback and forecast lengths.

Benchmark

Given the significant value to researchers and practitioners, we provide a summary of concurrent LTSMs:

We also establish the first zero-shot benchmark to measure LTSMs as a general-purpose forecaster.

It should be noticed that the zero-shot performance of concurrent Large Time Series models is still lagged behind large models based on few-shot fine-tuning or end-to-end training (similar to the challenges GPT-3 faced in 2020).

This is why we hightlight the few-shot ability of LTSMs instead of making zero-shot predictions on real-world application now. It is a long but promising direction to develop large models for ZSF.

Citation

If you find this repo helpful, please cite our paper.

@inproceedings{liutimer,
  title={Timer: Generative Pre-trained Transformers Are Large Time Series Models},
  author={Liu, Yong and Zhang, Haoran and Li, Chenyu and Huang, Xiangdong and Wang, Jianmin and Long, Mingsheng},
  booktitle={Forty-first International Conference on Machine Learning}
}

Acknowledgement

We appreciate the following GitHub repos a lot for their valuable code and efforts.

• Time-Series-Library (https://github.com/thuml/Time-Series-Library)

Contact

If you have any questions or want to use the code, feel free to contact:

• Yong Liu (liuyong21@mails.tsinghua.edu.cn)
• Haoran Zhang (z-hr20@mails.tsinghua.edu.cn)
• Chenyu Li (lichenyu20@mails.tsinghua.edu.cn)