After completing the training, the F1 score is not coming out as expected.

[hm-jeong]

Hello,

I am currently trying to reproduce the ATST-SED model in my environment.
I have completed training stages 1 and 2 on the DCASE2023 Task 4 baseline at the link below.
However, the F1 score turned out to be much lower than expected.

Event based metrics (onset-offset)
========================================
  Evaluated length                  : 8608.99 sec
  Evaluated files                   : 962 
  Evaluate onset                    : True 
  Evaluate offset                   : True 
  T collar                          : 200.00 ms
  Offset (length)                   : 20.00 %

  Overall metrics (micro-average)
  ======================================
  F-measure
    F-measure (F1)                  : 6.81 %
    Precision                       : 11.03 %
    Recall                          : 4.93 %
  Error rate
    Error rate (ER)                 : 1.34 
    Substitution rate               : 0.00 
    Deletion rate                   : 0.95 
    Insertion rate                  : 0.39 

  Class-wise average metrics (macro-average)
  ======================================
  F-measure
    F-measure (F1)                  : 15.88 %
    Precision                       : 20.36 %
    Recall                          : 13.52 %
  Error rate
    Error rate (ER)                 : 1.24 
    Deletion rate                   : 0.86 
    Insertion rate                  : 0.37 
  

  Class-wise metrics
  ======================================
    Event label  | Nref    Nsys  | F        Pre      Rec    | ER       Del      Ins    |
    ------------ | -----   ----- | ------   ------   ------ | ------   ------   ------ |
    Dishes       | 496     243   | 4.9%     7.4%     3.6%   | 1.42     0.96     0.45   |
    Electric_s.. | 51      30    | 37.0%    50.0%    29.4%  | 1.00     0.71     0.29   |
    Frying       | 87      58    | 40.0%    50.0%    33.3%  | 1.00     0.67     0.33   |
    Blender      | 81      19    | 2.0%     5.3%     1.2%   | 1.21     0.99     0.22   |
    Speech       | 1437    637   | 2.9%     4.7%     2.1%   | 1.40     0.98     0.42   |
    Cat          | 305     46    | 1.7%     6.5%     1.0%   | 1.13     0.99     0.14   |
    Dog          | 477     179   | 3.4%     6.1%     2.3%   | 1.33     0.98     0.35   |
    Alarm_bell.. | 337     166   | 2.8%     4.2%     2.1%   | 1.45     0.98     0.47   |
    Vacuum_cle.. | 75      70    | 48.3%    50.0%    46.7%  | 1.00     0.53     0.47   |
    Running_wa.. | 185     129   | 15.9%    19.4%    13.5%  | 1.43     0.86     0.56   |

Could this issue be caused by an incorrect dataset path in the config file?

I would appreciate your assistance.

Thank you.

https://github.com/DCASE-REPO/DESED_task/tree/master/recipes/dcase2023_task4_baseline

[SaoYear]

Could you please also provide the PSDS scores?

[hm-jeong]

Thank you for your quick answer :) Here are the PSDS scores.

scenario1

scenario2

[SaoYear]

It seems that the system is not working. There are several things you can do:

1. Reproduce the inference results with the pretrained ckpt file.
2. Check the sampling rate of the input audio files.
3. Logging the number of utterances in each dataset. The weak-labeled dataset ~1100 strong-real ~3000 synthetic ~10000 unlabeled ~14400

[SaoYear]

Hi, did you solve your problem?