Time Series Forecasting using Transformers - Thesis Repository

Repository containing my Master Thesis for the M.Sc. Big Data Analytics, titled "Time Series Forecasting using Transformers".

• Abstract: This repository contains the experimental work developed that has explored the usage of Transformer models for time-series forecasting. In particular, 5 different code notebooks host experiments with the Electricity and Traffic time-series benchmark datasets and a series of baseline classical models, together with the DeepAR deep model and two popular Transformer architectures: the Informer and the Temporal Fusion Transformer.
• Author: Andrés Carrillo López
• Supervisor: Pablo Martínez Olmos

---

Repository Folder Structure:

• img folder: contains images used in this README document and other relevant images.
• code_notebooks folder: contains Python code and notebooks (.ipynb) used for the experiments.
• data folder: contains the benchmark (public) datasets used in the experiments. The complete processed ones are compressed in bzip2.

Trained models:

For those models that training took considerable time, their trained checkpoint-model (heavy) files can be found here available for download.

---

Test Notebooks:

Important note: to view notebooks from these links, you must access them through a UC3M mail account. If you do not have a UC3M account, or simply want to open the notebooks in a local jupyter environment, download the notebooks from the code_notebooks folder and open them in a Google Colaboratory environment.

• Benchmark datasets (Traffic and Electricity) Retrieval and Aggregation:
• Classical forecast methods Tests:
• Temporal Fusion Transformer (TFT) Tests::
• Informer Tests:
• DeepAR Tests:

---

Current progress and milestones (WIP):

• Benchmark Datasets (Traffic, Electricity) Datasets retrieval and preprocessing.
• Tests Forecasting using Temporal Fusion Transformer (TFT) architecture.
• Tests Forecasting using DeepAR architecture.
• Tests Forecasting using Informer Transformer architecture.
• Tests Forecasting using classical methods ((S)ARIMA, SES and Holt-Winters).
• Ensure proper comparability between final error metrics in models/datasets.
• Provide links for experiment notebooks in GoogleColab playground notebooks.
• Final review / decoration of Test notebooks.
• Master Thesis (Current progress: 100%).

---

Some extracted results:

• Benchmark results (Traffic and Electricity datasets, 1-day and 1-week forecast scenarios) Mean Absolute Errors:

	ARIMA	SES	Holt-Winters	DeepAR	Informer	TFT
Traffic (1-day)	0.0188	0.0640	0.0236	0.0161	0.01050	0.0099
Traffic (1-week)	0.0198	0.0757	0.0336	0.0147	0.01251	0.0065
Electricity (1-day)	84.2201	307.4311	75.6006	88.3992	69.8323	36.0599
Electricity (1-week)	165.0958	308.4780	98.3879	64.1848	102.0774	49.5773

• Some images from notebook tests so far: sample plots extracted from the TFT model forecasts, with attention and feature relevances plots: