Question about the parameters #3

KcAcoZhang · 2020-12-29T13:12:55Z

Hello, can I ask why you set the batch_size=7919, and whether the z_size means latent space or not? Did you use the down-sampling rate mentioned in the article USAD? Thanks for your answer.

manigalati · 2020-12-29T15:46:21Z

Hello! The unusual value for the batch size aimed only to speed up the training process, I left it like that but you should be free to change it affecting only the training time. The z_size is actually the multiplication between the window size = 12 and the latent size = 10, and it indeed corresponds to the actual dimension of the latent space. I did not implement the experiments regarding the down-sampling, sorry.

KcAcoZhang · 2020-12-30T01:50:57Z

OK, thanks for your help.

KcAcoZhang · 2020-12-31T03:08:49Z

Hello, I have another question. I notice that you use the windows_normal_test+windows_attack to be the test_loader, but in the SWaT dataset, it has Normal/Attack label and you drop it, and not all the windows_attack data is abnormal. Thanks for your answer.

manigalati · 2020-12-31T05:56:15Z

Hello again :)
This is definitely an interesting point. The truth is that I reimplemented the code by myself without a strict supervision of the author of the paper, and since he sent me only a simple scratch of his implementation with dataloaders already implemented, I didn't double check the dataset and the labels. So you can be right, if that is the case in the next days I will correct the mistake. Let me know if you try before and get better results.

KcAcoZhang · 2020-12-31T06:36:00Z

Hi, I have just read the paper and studied your code, then I found out the question. Until now I still not try to do it. I'm looking forward to your correction and I will try to do it now.

severous · 2021-01-23T14:53:18Z

I also notice this issue. and I try to do it.but the permence of model is very pool.I am confused about it.

severous · 2021-01-23T15:43:20Z

My implementation is here

mariazuluaga · 2021-02-17T13:51:12Z

Hi @severous what do you mean by permence?

meihuameii · 2021-06-28T09:52:25Z

I have the same problem why all attack windows are abnormal?

JulienAu · 2021-06-28T11:58:42Z

I have the same problem why all attack windows are abnormal?

This problem has been solved in this implementation.
https://github.com/robustml-eurecom/usad

Thanks

meihuameii · 2021-06-29T02:53:55Z

I have the same problem why all attack windows are abnormal?

This problem has been solved in this implementation.
https://github.com/robustml-eurecom/usad

Thanks

Hello, thanks for your reply! I use the new code to do anomaly detection but I get bad results in f1_score. It' performance is not as good as the original paper.

mariazuluaga · 2021-07-01T07:41:05Z

Hi,
Could you please provide us with some more details regarding your issue? Which dataset?

I have the same problem why all attack windows are abnormal?

This problem has been solved in this implementation.
https://github.com/robustml-eurecom/usad
Thanks

Hello, thanks for your reply! I use the new code to do anomaly detection but I get bad results in f1_score. It' performance is not as good as the original paper.

meihuameii · 2021-07-02T02:53:54Z

Hi,
Could you please provide us with some more details regarding your issue? Which dataset?

I have the same problem why all attack windows are abnormal?

This problem has been solved in this implementation.
https://github.com/robustml-eurecom/usad
Thanks

Hello, thanks for your reply! I use the new code to do anomaly detection but I get bad results in f1_score. It' performance is not as good as the original paper.

I have solved this problem！The original paper selected the best F1_score, I use this way to achieve a similar result in SWaT. Thanks!

soemthlng · 2021-09-28T01:53:52Z

@meihuameii Hello, can you explain how to achieve a similar result in SWaT?

hanhuili · 2021-10-01T05:17:14Z

Hi,
Could you please provide us with some more details regarding your issue? Which dataset?

I have the same problem why all attack windows are abnormal?

This problem has been solved in this implementation.
https://github.com/robustml-eurecom/usad
Thanks

Hello, thanks for your reply! I use the new code to do anomaly detection but I get bad results in f1_score. It' performance is not as good as the original paper.

I have solved this problem！The original paper selected the best F1_score, I use this way to achieve a similar result in SWaT. Thanks!

Yet I select the best F1_score, it is about 74%, which is still far below that in the paper

finloop · 2021-10-04T05:25:14Z

@meihuameii Hello, can you explain how to achieve a similar result in SWaT?

Hi,
I've stumbled upon the same problem and found that increasing threshold can achieve similar results. Here are parameters and scores that I used on one of the runs:

window_size = 20 
BATCH_SIZE =  1000
N_EPOCHS = 50
hidden_size = 20

Then I increased threshold to 10 and found the predictions with:

threshold = 10
y_pred_ = np.zeros(y_pred.shape[0])
y_pred_[y_pred >= threshold] = 1

Classification report:

              precision    recall  f1-score   support

         0.0       0.95      0.99      0.97    394613
         1.0       0.88      0.63      0.74     55286

    accuracy                           0.94    449899
   macro avg       0.92      0.81      0.85    449899
weighted avg       0.94      0.94      0.94    449899

soemthlng · 2021-10-05T01:23:38Z

@meihuameii Hello, can you explain how to achieve a similar result in SWaT?

Hi, I've stumbled upon the same problem and found that increasing threshold can achieve similar results. Here are parameters and scores that I used on one of the runs:
window_size = 20 
BATCH_SIZE =  1000
N_EPOCHS = 50
hidden_size = 20
Then I increased threshold to 10 and found the predictions with:
threshold = 10
y_pred_ = np.zeros(y_pred.shape[0])
y_pred_[y_pred >= threshold] = 1
Classification report:
              precision    recall  f1-score   support

         0.0       0.95      0.99      0.97    394613
         1.0       0.88      0.63      0.74     55286

    accuracy                           0.94    449899
   macro avg       0.92      0.81      0.85    449899
weighted avg       0.94      0.94      0.94    449899

Hi,
Thank you for sharing your research. Here is my best score and parameters.
I used StandardScaler for preprocessing and I used y_test y_test=np.concatenate([np.zeros(windows_normal_test.shape[0]), np.ones(windows_attack.shape[0])]) instead of y_test = [1.0 if (np.sum(window) > 0) else 0 for window in windows_labels ]

window_size = 8 BATCH_SIZE = 7919 N_EPOCHS = 200 hidden_size = 20
I checked this score using sklearn.metrics
recall 0.8068351296145237
precision 0.8836299019011222
f1 0.8434881918763455

and did you mean use threshold and prediction in this way?
y_pred=np.concatenate([torch.stack(results[:-1]).flatten().detach().cpu().numpy(), results[-1].flatten().detach().cpu().numpy()]) threshold = 10 y_pred_ = np.zeros(y_pred.shape[0]) y_pred_[y_pred >= threshold] = 1

Im my case, i got recall 0.0852990924871808 precision 1.0 f1 0.1571900190051773

finloop · 2021-10-05T12:56:58Z

and did you mean use threshold and prediction in this way?

@soemthlng Yes, I increased threshold to trade of recall for f1 (or so I thought, I'm still learning). In my case the score before increasing the threshold was similar to:

              precision    recall  f1-score   support

         0.0       0.95      0.72      0.82    394901
         1.0       0.26      0.72      0.38     55006

    accuracy                           0.72    449907
   macro avg       0.61      0.72      0.60    449907
weighted avg       0.86      0.72      0.76    449907

How did you select threshold?

soemthlng · 2021-10-07T01:54:01Z

@finloop
In this case, recall 0.0852990924871808 precision 1.0 f1 0.1571900190051773 I select threshold to 10, but score is not good.

This is my code.

threshold = 10
y_pred_ = np.zeros(y_pred.shape[0])
y_pred_[y_pred >= threshold] = 1
y_test=np.concatenate([np.zeros(windows_normal_test.shape[0]), np.ones(windows_attack.shape[0])])

print("recall", recall_score(y_test,y_pred_))
print("precision", precision_score(y_test, y_pred_))
print("f1", f1_score(y_test, y_pred_))

I have 2 questions.

Did you modify y_test ?
Did you change the preprocessor function? I used StandardScaler instead of MinMaxScaler.

finloop · 2021-10-07T18:15:29Z

@soemthlng

I did not modify the y_test (I used the one provided in the repo).
Yes I used StandardScaler too.

You can check out the whole USAD.ipynb file.

I think I found the issue:

This line in your code could cause the divergence in scores:

y_test=np.concatenate([np.zeros(windows_normal_test.shape[0]), np.ones(windows_attack.shape[0])])

This line creates a long array of zeros and ones like [0,0,0,0 ... 1,1,1,1], it doesn't take into account that not all windows in windows_attack are 1.

This is what y_test looks like for me:

Those are my predictions:

This is what I think your y_test looks like:
!
I cannot recreate it because windows_normal_test(I assumed it's small) is not present in the original notebook. This would check out with your results:

#threshold=10
y_test=np.concatenate([np.ones(windows_attack.shape[0])])
plt.plot(y_test)
plt.ylim([0,1.5])
print(sklearn.metrics.classification_report(y_test, y_pred_))

              precision    recall  f1-score   support

         0.0       0.00      0.00      0.00         0
         1.0       1.00      0.09      0.16    449899

    accuracy                           0.09    449899
   macro avg       0.50      0.04      0.08    449899
weighted avg       1.00      0.09      0.16    449899

Could you show me your notebook?

soemthlng · 2021-10-12T06:43:25Z

@finloop

I run this code in 'Ununtu 16.04 Server, so I do not have notebook.

This is my code of y_test and y_pred
y_pred=np.concatenate([torch.stack(results[:-1]).flatten().detach().cpu().numpy(), results[-1].flatten().detach().cpu().numpy()])

y_test=np.concatenate([np.zeros(windows_normal_test.shape[0]), np.ones(windows_attack.shape[0])])

and this is my y_test figure

Did not modify the y_test mean that you use this?
y_test = [1.0 if (np.sum(window) > 0) else 0 for window in windows_labels ]
Could you share your full notebook?
I’ll use it only for study.

finloop · 2021-10-13T09:44:49Z

@soemthlng

Did not modify the y_test mean that you use this?
y_test = [1.0 if (np.sum(window) > 0) else 0 for window in windows_labels ]

Yes :)

Could you share your full notebook?

Sure. Here you go: Link to notebook. https://github.com/finloop/usad/blob/dev/USAD.ipynb

soemthlng · 2021-10-14T03:13:07Z

@finloop
After test your code, I am confused because of the metrics.

this is classification_report's result

                       precision    recall    f1-score   support

     0.0              0.95            0.99      0.97        394613
     1.0              0.91            0.63      0.75        55286

accuracy                                        0.95        449899
macro avg             0.93            0.81      0.86        449899
weighted avg          0.95            0.95      0.94        449899

I think 1.0 class's f1-score is what we want.
Are you using 0.0 class as true value?
Thanks.

finloop · 2021-10-22T07:21:07Z

@soemthlng

I think 1.0 class's f1-score is what we want.

Yes It is what we want.

Are you using 0.0 class as true value?

What do you mean by true value?.
Class 0.0 is what we consider as normal data.
Class 1.0 - anomalies.

soemthlng · 2021-10-22T10:22:17Z

@finloop

I mean true value is the value that I want to find.

Previously, you said that you got results similar to the results presented in the paper.
Did you mean that the value was based on class 0.0?

finloop · 2021-10-22T12:46:38Z

@soemthlng

Did you mean that the value was based on class 0.0?

No. The f1, recall etc. was based on class 1.0. Also my results were slightly worse than the ones in the paper (check it out if you want, it's in README).

pzpzpzp · 2022-09-21T08:09:26Z

Hi, Could you please provide us with some more details regarding your issue? Which dataset?

I have the same problem why all attack windows are abnormal?

This problem has been solved in this implementation.
https://github.com/robustml-eurecom/usad
Thanks

Hello, thanks for your reply! I use the new code to do anomaly detection but I get bad results in f1_score. It' performance is not as good as the original paper.

Hello, have you reproduced the results from the original paper (F1 result without point-adjust on the SWAT dataset is 0.7917). I've been very distressed recently that I can't reproduce the results from my original paper. In my code, BATCH_ SIZE = 1024, N_ EPOCHS = 100, hidden_ Size = 20, window_ Size=10. Standard Scaler () or MinMaxScaler () were used for data preprocessing, and downsampling was used in the code, with a downsampling rate of 5. The best results from these methods were only about 0.74. Do you have any skills to make the results similar to those of the original paper? I'd love to wait for your reply! Thank you very much.

dadodimauro · 2023-03-01T16:31:17Z

Hello,
Has anyone managed to get the results of the paper? if so how?

Thanks!

hisong182 · 2023-03-09T10:20:43Z

y_test = [1.0 if (np.sum(window) > 0) else 0 for window in windows_labels ]

Hello, sorry to bother you, may I ask why you set the threshold to 0, after I set the result 1.0 acc value is 0

finloop · 2023-03-10T14:11:06Z

Hello, sorry to bother you, may I ask why you set the threshold to 0, after I set the result 1.0 acc value is 0
@hisong182

A window is considered anomalous if it contains at least one anomaly. Checkout my notebook: https://github.com/finloop/usad/blob/dev/USAD.ipynb

hisong182 · 2023-03-12T09:32:57Z

A window is considered anomalous if it contains at least one anomaly. Checkout my notebook:

Thank you for your answer, before I set the wrong data normalization to MinMaxScaler, after the change the accuracy increased a lot

Donek1 · 2024-02-22T14:48:13Z

Thanks. How to get the best result on wadi dataset?

Jhonzhang · 2024-07-05T09:28:48Z

Hi, Could you please provide us with some more details regarding your issue? Which dataset?

I have the same problem why all attack windows are abnormal?

This problem has been solved in this implementation.
https://github.com/robustml-eurecom/usad
Thanks

Hello, thanks for your reply! I use the new code to do anomaly detection but I get bad results in f1_score. It' performance is not as good as the original paper.

Hello, have you reproduced the results from the original paper (F1 result without point-adjust on the SWAT dataset is 0.7917). I've been very distressed recently that I can't reproduce the results from my original paper. In my code, BATCH_ SIZE = 1024, N_ EPOCHS = 100, hidden_ Size = 20, window_ Size=10. Standard Scaler () or MinMaxScaler () were used for data preprocessing, and downsampling was used in the code, with a downsampling rate of 5. The best results from these methods were only about 0.74. Do you have any skills to make the results similar to those of the original paper? I'd love to wait for your reply! Thank you very much.

I have the same questions too. I will check out the new codes which was uploaded by @finloop https://github.com/finloop/usad/blob/dev/USAD.ipynb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Question about the parameters #3

Question about the parameters #3

KcAcoZhang commented Dec 29, 2020

manigalati commented Dec 29, 2020

KcAcoZhang commented Dec 30, 2020

KcAcoZhang commented Dec 31, 2020

manigalati commented Dec 31, 2020

KcAcoZhang commented Dec 31, 2020

severous commented Jan 23, 2021

severous commented Jan 23, 2021

mariazuluaga commented Feb 17, 2021

meihuameii commented Jun 28, 2021

JulienAu commented Jun 28, 2021

meihuameii commented Jun 29, 2021 •

edited

Loading

mariazuluaga commented Jul 1, 2021

meihuameii commented Jul 2, 2021

soemthlng commented Sep 28, 2021

hanhuili commented Oct 1, 2021

finloop commented Oct 4, 2021

soemthlng commented Oct 5, 2021 •

edited

Loading

finloop commented Oct 5, 2021

soemthlng commented Oct 7, 2021 •

edited

Loading

finloop commented Oct 7, 2021

soemthlng commented Oct 12, 2021 •

edited

Loading

finloop commented Oct 13, 2021

soemthlng commented Oct 14, 2021 •

edited

Loading

finloop commented Oct 22, 2021

soemthlng commented Oct 22, 2021

finloop commented Oct 22, 2021

pzpzpzp commented Sep 21, 2022

dadodimauro commented Mar 1, 2023

hisong182 commented Mar 9, 2023

finloop commented Mar 10, 2023

hisong182 commented Mar 12, 2023

Donek1 commented Feb 22, 2024

Jhonzhang commented Jul 5, 2024

Question about the parameters #3

Question about the parameters #3

Comments

KcAcoZhang commented Dec 29, 2020

manigalati commented Dec 29, 2020

KcAcoZhang commented Dec 30, 2020

KcAcoZhang commented Dec 31, 2020

manigalati commented Dec 31, 2020

KcAcoZhang commented Dec 31, 2020

severous commented Jan 23, 2021

severous commented Jan 23, 2021

mariazuluaga commented Feb 17, 2021

meihuameii commented Jun 28, 2021

JulienAu commented Jun 28, 2021

meihuameii commented Jun 29, 2021 • edited Loading

mariazuluaga commented Jul 1, 2021

meihuameii commented Jul 2, 2021

soemthlng commented Sep 28, 2021

hanhuili commented Oct 1, 2021

finloop commented Oct 4, 2021

soemthlng commented Oct 5, 2021 • edited Loading

finloop commented Oct 5, 2021

soemthlng commented Oct 7, 2021 • edited Loading

finloop commented Oct 7, 2021

I think I found the issue:

soemthlng commented Oct 12, 2021 • edited Loading

I run this code in 'Ununtu 16.04 Server, so I do not have notebook.

finloop commented Oct 13, 2021

soemthlng commented Oct 14, 2021 • edited Loading

finloop commented Oct 22, 2021

soemthlng commented Oct 22, 2021

finloop commented Oct 22, 2021

pzpzpzp commented Sep 21, 2022

dadodimauro commented Mar 1, 2023

hisong182 commented Mar 9, 2023

finloop commented Mar 10, 2023

hisong182 commented Mar 12, 2023

Donek1 commented Feb 22, 2024

Jhonzhang commented Jul 5, 2024

meihuameii commented Jun 29, 2021 •

edited

Loading

soemthlng commented Oct 5, 2021 •

edited

Loading

soemthlng commented Oct 7, 2021 •

edited

Loading

soemthlng commented Oct 12, 2021 •

edited

Loading

soemthlng commented Oct 14, 2021 •

edited

Loading