Skip to content

Commit

Permalink
Merge pull request #501 from WenjieDu/dev
Browse files Browse the repository at this point in the history
Add TimeMixer
  • Loading branch information
WenjieDu authored Sep 3, 2024
2 parents dd8c553 + 8b4675e commit 2d702aa
Show file tree
Hide file tree
Showing 14 changed files with 1,262 additions and 6 deletions.
Binary file added .idea/icon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,8 +116,9 @@ The paper references and links are all listed at the bottom of this file.
| **Type** | **Algo** | **IMPU** | **FORE** | **CLAS** | **CLUS** | **ANOD** | **Year - Venue** |
|:--------------|:---------------------------------------------------------------------------------------------------------------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|:-------------------|
| LLM | <a href="https://time-series.ai"><img src="https://time-series.ai/static/figs/robot.svg" width="26px"> Time-Series.AI</a> [^36] |||||| `Later in 2024` |
| Neural Net | ImputeFormer🧑‍🔧[^34] || | | | | `2024 - KDD` |
| Neural Net | TimeMixer[^37] || | | | | `2024 - ICLR` |
| Neural Net | iTransformer🧑‍🔧[^24] || | | | | `2024 - ICLR` |
| Neural Net | ImputeFormer🧑‍🔧[^34] || | | | | `2024 - KDD` |
| Neural Net | SAITS[^1] || | | | | `2023 - ESWA` |
| Neural Net | FreTS🧑‍🔧[^23] || | | | | `2023 - NeurIPS` |
| Neural Net | Koopa🧑‍🔧[^29] || | | | | `2023 - NeurIPS` |
Expand Down Expand Up @@ -320,7 +321,7 @@ By committing your code, you'll
[pypots/imputation/template](https://github.com/WenjieDu/PyPOTS/tree/main/pypots/imputation/template)) to quickly start;
2. become one of [PyPOTS contributors](https://github.com/WenjieDu/PyPOTS/graphs/contributors) and
be listed as a volunteer developer [on the PyPOTS website](https://pypots.com/about/#volunteer-developers);
3. get mentioned in our [release notes](https://github.com/WenjieDu/PyPOTS/releases);
3. get mentioned in PyPOTS [release notes](https://github.com/WenjieDu/PyPOTS/releases);

You can also contribute to PyPOTS by simply staring🌟 this repo to help more people notice it.
Your star is your recognition to PyPOTS, and it matters!
Expand Down Expand Up @@ -395,3 +396,4 @@ PyPOTS community is open, transparent, and surely friendly. Let's work together
[^36]: Project Gungnir, the world 1st LLM for time-series multitask modeling, will meet you soon. 🚀 Missing values and variable lengths in your datasets?
Hard to perform multitask learning with your time series? Not problems no longer. We'll open application for public beta test recently ;-) Follow us, and stay tuned!
<a href="https://time-series.ai"><img src="https://time-series.ai/static/figs/robot.svg" width="20px" align="center"> Time-Series.AI</a>
[^37]: Wang, S., Wu, H., Shi, X., Hu, T., Luo, H., Ma, L., ... & ZHOU, J. (2024). [TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting](https://openreview.net/forum?id=7oLshfEIC2). *ICLR 2024*
6 changes: 4 additions & 2 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,9 @@ PyPOTS当前支持多变量POTS数据的插补,预测,分类,聚类以及
| **类型** | **算法** | **插补** | **预测** | **分类** | **聚类** | **异常检测** | **年份 - 刊物** |
|:--------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------|:------:|:------:|:------:|:------:|:--------:|:-------------------|
| LLM | <a href="https://time-series.ai"><img src="https://time-series.ai/static/figs/robot.svg" width="26px"> Time-Series.AI</a> [^36] |||||| `Later in 2024` |
| Neural Net | ImputeFormer🧑‍🔧[^34] || | | | | `2024 - KDD` |
| Neural Net | TimeMixer[^37] || | | | | `2024 - ICLR` |
| Neural Net | iTransformer🧑‍🔧[^24] || | | | | `2024 - ICLR` |
| Neural Net | ImputeFormer🧑‍🔧[^34] || | | | | `2024 - KDD` |
| Neural Net | SAITS[^1] || | | | | `2023 - ESWA` |
| Neural Net | FreTS🧑‍🔧[^23] || | | | | `2023 - NeurIPS` |
| Neural Net | Koopa🧑‍🔧[^29] || | | | | `2023 - NeurIPS` |
Expand Down Expand Up @@ -292,7 +293,7 @@ year={2023},
[pypots/imputation/template](https://github.com/WenjieDu/PyPOTS/tree/main/pypots/imputation/template))快速启动你的开发;
2. 成为[PyPOTS贡献者](https://github.com/WenjieDu/PyPOTS/graphs/contributors)之一,
并在[PyPOTS网站](https://pypots.com/about/#volunteer-developers)上被列为志愿开发者;
3. 在我们发布新版本的[更新日志](https://github.com/WenjieDu/PyPOTS/releases)中被提及;
3. 在PyPOTS发布新版本的[更新日志](https://github.com/WenjieDu/PyPOTS/releases)中被提及;

你也可以通过为该项目设置星标🌟,帮助更多人关注它。你的星标🌟既是对PyPOTS的认可,也是对PyPOTS发展所做出的重要贡献!

Expand Down Expand Up @@ -364,3 +365,4 @@ PyPOTS社区是一个开放、透明、友好的社区,让我们共同努力
[^35]: Bai, S., Kolter, J. Z., & Koltun, V. (2018). [An empirical evaluation of generic convolutional and recurrent networks for sequence modeling](https://arxiv.org/abs/1803.01271). *arXiv 2018*.
[^36]: Gungnir项目,世界上第一个时间序列多任务大模型,将很快与大家见面。🚀 数据集存在缺少值且样本长短不一?多任务建模场景困难?都不再是问题,让我们的大模型来帮你解决。我们将在近期开放公测申请 ;-) 关注我们,敬请期待!
<a href="https://time-series.ai"><img src="https://time-series.ai/static/figs/robot.svg" width="20px" align="center"> Time-Series.AI</a>
[^37]: Wang, S., Wu, H., Shi, X., Hu, T., Luo, H., Ma, L., ... & ZHOU, J. (2024). [TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting](https://openreview.net/forum?id=7oLshfEIC2). *ICLR 2024*
6 changes: 4 additions & 2 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -133,10 +133,12 @@ The paper references are all listed at the bottom of this readme file.
+----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+
| Type | Algorithm | IMPU | FORE | CLAS | CLUS | ANOD | Year - Venue |
+================+===========================================================+======+======+======+======+======+=======================+
| Neural Net | ImputeFormer :cite:`nie2024imputeformer` || | | | | ``2024 - KDD`` |
| Neural Net | TimeMixer :cite:`wang2024timemixer` || | | | | ``2024 - ICLR`` |
+----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+
| Neural Net | iTransformer🧑‍🔧 :cite:`liu2024itransformer` || | | | | ``2024 - ICLR`` |
+----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+
| Neural Net | ImputeFormer :cite:`nie2024imputeformer` || | | | | ``2024 - KDD`` |
+----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+
| Neural Net | SAITS :cite:`du2023SAITS` || | | | | ``2023 - ESWA`` |
+----------------+-----------------------------------------------------------+------+------+------+------+------+-----------------------+
| Neural Net | FreTS🧑‍🔧 :cite:`yi2023frets` || | | | | ``2023 - NeurIPS`` |
Expand Down Expand Up @@ -333,7 +335,7 @@ By committing your code, you'll
`pypots/imputation/template <https://github.com/WenjieDu/PyPOTS/tree/main/pypots/imputation/template>`_) to quickly start;
2. become one of `PyPOTS contributors <https://github.com/WenjieDu/PyPOTS/graphs/contributors>`_ and
be listed as a volunteer developer `on the PyPOTS website <https://pypots.com/about/#volunteer-developers>`_;
3. get mentioned in our `release notes <https://github.com/WenjieDu/PyPOTS/releases>`_;
3. get mentioned in PyPOTS `release notes <https://github.com/WenjieDu/PyPOTS/releases>`_;

You can also contribute to PyPOTS by simply staring🌟 this repo to help more people notice it.
Your star is your recognition to PyPOTS, and it matters!
Expand Down
9 changes: 9 additions & 0 deletions docs/pypots.imputation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,15 @@ pypots.imputation.transformer
:show-inheritance:
:inherited-members:

pypots.imputation.timemixer
------------------------------------

.. automodule:: pypots.imputation.timemixer
:members:
:undoc-members:
:show-inheritance:
:inherited-members:

pypots.imputation.imputeformer
------------------------------------

Expand Down
2 changes: 2 additions & 0 deletions pypots/imputation/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
from .grud import GRUD
from .stemgnn import StemGNN
from .imputeformer import ImputeFormer
from .timemixer import TimeMixer

# naive imputation methods
from .locf import LOCF
Expand Down Expand Up @@ -75,6 +76,7 @@
"GRUD",
"StemGNN",
"ImputeFormer",
"TimeMixer",
# naive imputation methods
"LOCF",
"Mean",
Expand Down
24 changes: 24 additions & 0 deletions pypots/imputation/timemixer/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
"""
The package including the modules of TimeMixer.
Refer to the paper
`Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y. Zhang, and Jun Zhou.
"TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting".
In ICLR 2024.
<https://openreview.net/pdf?id=7oLshfEIC2>`_
Notes
-----
This implementation is inspired by the official one https://github.com/kwuking/TimeMixer
"""

# Created by Wenjie Du <wenjay.du@gmail.com>
# License: BSD-3-Clause


from .model import TimeMixer

__all__ = [
"TimeMixer",
]
83 changes: 83 additions & 0 deletions pypots/imputation/timemixer/core.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
"""
"""

# Created by Wenjie Du <wenjay.du@gmail.com>
# License: BSD-3-Clause

import torch.nn as nn

from ...nn.functional import (
nonstationary_norm,
nonstationary_denorm,
)
from ...nn.modules.timemixer import BackboneTimeMixer
from ...utils.metrics import calc_mse


class _TimeMixer(nn.Module):
def __init__(
self,
n_layers,
n_steps,
n_features,
d_model,
d_ffn,
dropout,
top_k,
channel_independence,
decomp_method,
moving_avg,
downsampling_layers,
downsampling_window,
apply_nonstationary_norm: bool = False,
):
super().__init__()

self.apply_nonstationary_norm = apply_nonstationary_norm

self.model = BackboneTimeMixer(
task_name="imputation",
n_steps=n_steps,
n_features=n_features,
n_pred_steps=None,
n_pred_features=n_features,
n_layers=n_layers,
d_model=d_model,
d_ffn=d_ffn,
dropout=dropout,
channel_independence=channel_independence,
decomp_method=decomp_method,
top_k=top_k,
moving_avg=moving_avg,
downsampling_layers=downsampling_layers,
downsampling_window=downsampling_window,
downsampling_method="avg",
use_future_temporal_feature=False,
)

def forward(self, inputs: dict, training: bool = True) -> dict:
X, missing_mask = inputs["X"], inputs["missing_mask"]

if self.apply_nonstationary_norm:
# Normalization from Non-stationary Transformer
X, means, stdev = nonstationary_norm(X, missing_mask)

# TimesMixer processing
dec_out = self.model.imputation(X, None)

if self.apply_nonstationary_norm:
# De-Normalization from Non-stationary Transformer
dec_out = nonstationary_denorm(dec_out, means, stdev)

imputed_data = missing_mask * X + (1 - missing_mask) * dec_out
results = {
"imputed_data": imputed_data,
}

if training:
# `loss` is always the item for backward propagating to update the model
loss = calc_mse(dec_out, inputs["X_ori"], inputs["indicating_mask"])
results["loss"] = loss

return results
24 changes: 24 additions & 0 deletions pypots/imputation/timemixer/data.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
"""
Dataset class for the imputation model TimeMixer.
"""

# Created by Wenjie Du <wenjay.du@gmail.com>
# License: BSD-3-Clause

from typing import Union

from ..saits.data import DatasetForSAITS


class DatasetForTimeMixer(DatasetForSAITS):
"""Actually TimeMixer uses the same data strategy as SAITS, needs MIT for training."""

def __init__(
self,
data: Union[dict, str],
return_X_ori: bool,
return_y: bool,
file_type: str = "hdf5",
rate: float = 0.2,
):
super().__init__(data, return_X_ori, return_y, file_type, rate)
Loading

0 comments on commit 2d702aa

Please sign in to comment.