Reproducing results from WEASEL paper

[bkj]

Hi --

Do you have an example of using WEASEL to get results close to those reported in the paper? I'm playing around w/ using WEASEL as a featurizer on some of the UCR datasets, but my results aren't nearly as good as in the paper -- I'm guessing I'm not using the right hyperparameters. Any ideas?

Thanks!

[johannfaouzi]

Hi,

I created another repository where I compare the accuracies with the implementations in pyts and the accuracies reported in the original papers: pyts-repro.

More specifically about WEASEL, here is the corresponding notebook.

Please note that the default parameters in pyts are different from the ones used in the original implementation, especially regarding the window sizes:

• pyts: [0.1, 0.3, 0.5, 0.7, 0.9] (expressed as a percentage of the length of the time series),
• original implementation : from 2 to 350.

Another important hyper-parameter is the word size, which is chosen with cross-validation from 4 to 6 in the original implementation.

I hope that it helps a bit.

Johann

[bkj]

Thanks!

I assume the values for word and window size are tuned via cross validation in that notebook?

Edit: I also notice that the numbers you're getting there are almost all lower than the numbers reported in the paper. Why do you think that might be? The window size differences?

[johannfaouzi]

As mentioned in the introduction in the README, I did not perform an unbiased cross-validation to find the best values for the hyper-parameters (because I used the test set to find them), thus the numbers reported in the pyts column are the minimum of the results reported in the paper and the results obtained with a biased cross-validation.

What I strive to provide with this package is reusable code, with a lot of time spent on writing proper and readable code, and precise documentation. For this reason, I focus less on strict reproducibility of the original papers (several algorithms have a built-in cross-validation procedure, and I prefer not implementing it and letting the user do it on their own). If the performance with pyts is comparable with the performance reported in the paper, I think that it is enough (at least for now).