When running the with-future setting, for example:
python ./scripts/run_benchmark.py \ --config-path "rolling_forecast_config.json" \ --data-name-list "Sdwpfm1.csv" \ --strategy-args '{"horizon": 24, "target_channel": [-1]}' \ --adapter "transformer_adapter" \ --model-name "time_series_library.TemporalFusionTransformer" \ --model-hyper-params '{"batch_size": 32, "c_out": 7, "covariate_dim": 6, "d_ff": 512, "d_layers": 1, "d_model": 128, "data": "Sdwpfm1", "dec_in": 7, "dropout": 0.3, "e_layers": 2, "enc_in": 7, "factor": 3, "features": "MS", "horizon": 24, "label_len": 48, "lr": 0.001, "n_heads": 8, "norm": true, "num_epochs": 10, "patience": 3, "seq_len": 168}'
the original dataset columns are:[T2m, Sp, RelH, Wspd_w, Wdir_w, Tp, Patv]
Then forecast_fit() concatenates them as: [Patv, T2m, Sp, RelH, Wspd_w, Wdir_w, Tp]
Potential Bug
In TemporalFusionTransformer.forward(), the code reorders x_enc:
series_dim = x_enc.shape[-1] - x_dec.shape[-1] x_enc = torch.cat([x_enc[:, :, series_dim:], x_enc[:, :, :series_dim]], dim=-1)
So, this changes the order to: [T2m, Sp, RelH, Wspd_w, Wdir_w, Tp, Patv]
However, inside TFTEmbedding.forward(), the code again computes:
series_dim = x_enc.shape[-1] - x_dec.shape[-1] cov_x_mark = torch.cat([x_enc[:, :, series_dim:], x_dec], dim=-2)
After the reordering, x_enc[:, :, series_dim:] will select [Sp, RelH, Wspd_w, Wdir_w, Tp, Patv] as covariates, will make data leakage.
When running the with-future setting, for example:
python ./scripts/run_benchmark.py \ --config-path "rolling_forecast_config.json" \ --data-name-list "Sdwpfm1.csv" \ --strategy-args '{"horizon": 24, "target_channel": [-1]}' \ --adapter "transformer_adapter" \ --model-name "time_series_library.TemporalFusionTransformer" \ --model-hyper-params '{"batch_size": 32, "c_out": 7, "covariate_dim": 6, "d_ff": 512, "d_layers": 1, "d_model": 128, "data": "Sdwpfm1", "dec_in": 7, "dropout": 0.3, "e_layers": 2, "enc_in": 7, "factor": 3, "features": "MS", "horizon": 24, "label_len": 48, "lr": 0.001, "n_heads": 8, "norm": true, "num_epochs": 10, "patience": 3, "seq_len": 168}'the original dataset columns are:
[T2m, Sp, RelH, Wspd_w, Wdir_w, Tp, Patv]Then
forecast_fit()concatenates them as:[Patv, T2m, Sp, RelH, Wspd_w, Wdir_w, Tp]Potential Bug
In
TemporalFusionTransformer.forward(), the code reordersx_enc:series_dim = x_enc.shape[-1] - x_dec.shape[-1] x_enc = torch.cat([x_enc[:, :, series_dim:], x_enc[:, :, :series_dim]], dim=-1)So, this changes the order to:
[T2m, Sp, RelH, Wspd_w, Wdir_w, Tp, Patv]However, inside
TFTEmbedding.forward(), the code again computes:series_dim = x_enc.shape[-1] - x_dec.shape[-1] cov_x_mark = torch.cat([x_enc[:, :, series_dim:], x_dec], dim=-2)After the reordering,
x_enc[:, :, series_dim:]will select[Sp, RelH, Wspd_w, Wdir_w, Tp, Patv]as covariates, will make data leakage.