❓How to train/test EfficientAD without abnormal images

[Akihiko0123]

Thank you very much for sharing this great deep learning library for the anomaly detection.

I am trying to use EfficientAD with my original dataset, but I have some trouble to use it.

I thought EfficientAD is an unsupervised learning model and the input data of this model is normal images not abnormal images.

However, when I use my own dataset without writing abnormal_dir on the config.yaml, I received the following error.

    raise ConfigKeyError(f"Missing key {key!s}")
omegaconf.errors.ConfigAttributeError: Missing key abnormal_dir
    full_key: dataset.abnormal_dir
    object_type=dict

As for unsupervised learning models, I thought abnormal_dir which probably contain abnormal images is not necessary.

I would appreciate it if you could let me know on the followings.

1. Could you tell me if the input data of EfficientAD is just the normal images not abnormal images.

2. Since the default "task" in the config.yaml of EfficientAD is segmentation, is not the output of EfficientAD the anomaly score of each patch image?

3. Are mask images necessary as the input data for the segmentation task of EfficientAD?

Thank you.

[samet-akcay]

Hi @Akihiko0123, glad that you like the library!

Can you share your config file or python code to run the model? Abnormal directory is used to run the validation and test to get performance metrics. They are not needed during training.

[Akihiko0123]

Hi @samet-akcay,
I really appreciate you for your prompt reply!

I understand that Abnormal directory is used to run the validation and test to get performance metrics.
In that case, should I prepare a lot of abnormal data for the validation and test ?
I thought the abnormal images are used only during the inference phase to see the result.

My config file is as follows.
If there are items should be fixed, I would appreciate it if you could point them out.

dataset:
  name: original
  format: folder
  path: ../test_codes_anogan/
  normal_dir: train_sample # name of the folder containing normal images.
  abnormal_dir: null # name of the folder containing abnormal images.
  normal_test_dir: null # name of the folder containing normal test images.
  task: segmentation
  mask: null
  extensions: null
  train_batch_size: 1
  eval_batch_size: 16
  num_workers: 8
  image_size: 256 # dimensions to which images are resized (mandatory)
  center_crop: null # dimensions to which images are center-cropped after resizing (optional)
  normalization: imagenet # data distribution to which the images will be normalized: [none, imagenet]
  transform_config:
    train: null
    eval: null
  test_split_mode: none # options: [from_dir, synthetic]
  test_split_ratio: 0.2 # fraction of train images held out testing (usage depends on test_split_mode)
  val_split_mode: same_as_test # options: [same_as_test, from_test, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)

model:
  name: efficient_ad
  teacher_out_channels: 384
  model_size: small # options: [small, medium]
  lr: 0.0001
  weight_decay: 0.00001
  padding: false
  pad_maps: true # relevant for "padding: false", see EfficientAd in lightning_model.py
  normalization_method: min_max # options: [null, min_max, cdf] # min_max...最大1最小0に正規化

metrics:
  image:
    - F1Score
    - AUROC
  pixel:
    - F1Score
    - AUROC
  threshold:
    method: adaptive #options: [adaptive, manual]
    manual_image: null
    manual_pixel: null

visualization:
  show_images: False # show images on the screen
  save_images: True # save images to the file system
  log_images: False # log images to the available loggers (if any)
  image_save_path: null # path to which images will be saved
  mode: full # options: ["full", "simple"]

project:
  seed: 42
  path: ./results

logging:
  logger: [wandb] # options: [comet, tensorboard, wandb, csv] or combinations.
  log_graph: true # Logs the model graph to respective logger.

optimization:
  export_mode: null # options: torch, onnx, openvino
trainer:
  enable_checkpointing: true
  default_root_dir: null
  gradient_clip_val: 0
  gradient_clip_algorithm: norm
  num_nodes: 1
  devices: 1
  enable_progress_bar: true
  overfit_batches: 0.0
  track_grad_norm: -1
  check_val_every_n_epoch: 1
  fast_dev_run: false
  accumulate_grad_batches: 1
  max_epochs: 200
  min_epochs: null
  max_steps: 160000
  min_steps: null
  max_time: null
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  limit_test_batches: 1.0
  limit_predict_batches: 1.0
  val_check_interval: 1.0
  log_every_n_steps: 50
  accelerator: auto # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
  strategy: null
  sync_batchnorm: false
  precision: 32
  enable_model_summary: true
  num_sanity_val_steps: 0
  profiler: null
  benchmark: false
  deterministic: false
  reload_dataloaders_every_n_epochs: 0
  auto_lr_find: false
  replace_sampler_ddp: true
  detect_anomaly: false
  auto_scale_batch_size: false
  plugins: null
  move_metrics_to_cpu: false
  multiple_trainloader_mode: max_size_cycle

Thank you again.

[samet-akcay]

Thanks. I see that you are setting the validation set in your config.

  test_split_mode: none # options: [from_dir, synthetic]
  test_split_ratio: 0.2 # fraction of train images held out testing (usage depends on test_split_mode)
  val_split_mode: same_as_test # options: [same_as_test, from_test, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)

It is same_as_test. Despite test_split_mode being None, it probably fails, mainly due to validation split not being a None value

If you set val_split_mode to none, it should work. However, note that this will not run the adaptive thresholding and normalization properly. So you might need to find a manual threshold on your own.

Hope that helps.

[Akihiko0123]

Thank you so much for your help.

I set test_split_mode as none just to avoid an error.
When I set test_split_mode as from_dir, I received an error as follows.

  File "/mnt/project/science_intern/hirao/202311_S900007-KidneyPathology/hirao_work/anomalib/src/anomalib/data/folder.py", line 285, in __init__
    raise ValueError(
ValueError: Segmentation task requires mask directory if test_split_mode is from_dir. You could set test_split_mode to none or provide a mask directory.

I apologize for my persistence in asking, but could you let me know if I need to prepare the mask_dir and mask images for this task?
Thank you.

[samet-akcay]

yes, you are right, please ignore my previous reply. You could try something like this:

dataset:
  name: original
  format: folder
  root: ../test_codes_anogan/
  normal_dir: train_sample
  abnormal_dir: null
  normal_test_dir: null
  mask_dir: null
  extensions: null
  task: classification
  train_batch_size: 32
  eval_batch_size: 32
  num_workers: 8
  image_size: 256 # dimensions to which images are resized (mandatory)
  center_crop: null # dimensions to which images are center-cropped after resizing (optional)
  normalization: imagenet # data distribution to which the images will be normalized: [none, imagenet]
  transform_config:
    train: null
    eval: null
  test_split_mode: from_dir # options: [from_dir, synthetic]
  test_split_ratio: 0.2 # fraction of train images held out testing (usage depends on test_split_mode)
  val_split_mode: same_as_test # options: [same_as_test, from_test, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)

Since you don't have anomalous images you will probably see 0% performance at the end of the training. However, when you add some anomalous images to test the model, you should get some predictions. They may not be with the highest accuracy, but should give you an idea.

[Akihiko0123]

Thank you very much for your detailed information.
I will try using your config option.

As for the anomalous images, although I have several images, I cannot use them because the requirements of our goal is to train the model without anomalous images in the future.

Thank you again for your help.

[samet-akcay]

yeah, that makes sense. You could potentially have a look at the synthetic anomaly generation stuff as well

[Akihiko0123]

I really appreciate for your advice.
Since I am not sure on the synthetic anomaly generation things, I might need to check them in the future.

Although it is a supplementary question, would you mind letting me know on the followings ?

A.
Could you let me know how I should understand the output result of the classification?
I received some results after trying to train and test with the configuration file advised.
The results were like normal(53%) or anomalous(100%).
Since I thought EfficientAD outputs the anomalous score and the anomalous map, I would appreciate it if you could let me know how those figures calculated.

B.
I have the train data directory and validation data directory respectively.
However, according to the configuration file, since "test_split_ratio:0.2", is the test data is extracted from 20% of train data?
Also, is validation data extracted from 50% of train data considering "valid_split_ratio:0.5"?
If the above is not correct, I would appreciate it if you could tell me.

I apologize for bombarding you with questions, but I am grateful for your answers.
Thank you.