How to create metric visualisation when training

[lathashree01]

Hi I am training Patchcore model via engine

    engine = Engine(task=task, 
                    threshold=threshold,
                    image_metrics=image_metrics
                   )
   engine.train(datamodule=datamodule, model=model)

my images_metrics is ['Precision', 'Recall', 'AUPR']

I would like to generate ROC curves at the end of training. In previous version, I saw there was metrics visualiser callback but the latest does not, how can I still do it?

Thanks for the help

[lilruwu]

Hello, same problem here!

[samet-akcay]

Currently Visualizer is hard-coded. Though we are working on it to make it customisable from the engine.
You will need to manually add MetricVisualizer to here.

anomalib/src/anomalib/engine/engine.py

Lines 435 to 441 in 4b1e08c

	_callbacks.append(
	_VisualizationCallback(
	visualizers=ImageVisualizer(task=self.task, normalize=self.normalization == NormalizationMethod.NONE),
	save=True,
	root=self._cache.args["default_root_dir"] / "images",
	),
	)

So you would need something like this:

        _callbacks.append(
            _VisualizationCallback(
                visualizers=[
                    ImageVisualizer(task=self.task, normalize=self.normalization == NormalizationMethod.NONE),
                    MetricsVisualizer()
                ],
                save=True,
                root=self._cache.args["default_root_dir"] / "images",
            ),
        )

[lilruwu]

Is MetricsVisualizer() defined or should I define it? Cause I got this error:

NameError: name 'MetricsVisualizer' is not defined

Thank you!

[samet-akcay]

You need to import it like from anomalib.utils.visualization import MetricsVisualizer

[lilruwu]

Thank you! The images are stored in an special folder or where? Cause I have a folder in my configuration file for default_root_dir but there is nothing stored there.

Thank you so much and sorry for the inconveniences!

[samet-akcay]

can you share your training code? What metrics do you want to log?

[lilruwu]

Yes, here it is:

import os
import torch
# from pytorch_lightning import Trainer

from lightning.pytorch import Trainer
from lightning.pytorch.loggers import MLFlowLogger
from anomalib.engine import Engine
from anomalib.data import get_datamodule
from anomalib.models import get_model
from anomalib.callbacks import LoadModelCallback, get_callbacks
from anomalib.models.image.fastflow.lightning_model import Fastflow
from anomalib.models.components import AnomalyModule
from anomalib.metrics import AUROC

os.environ['TORCH_CUDA_ARCH_LIST'] = '7.5'

print('CUDA available devices:')
for i in range(torch.cuda.device_count()):
    print(torch.cuda.get_device_properties(i).name)

CONFIG_PATH = "config.yml"


engine, model, datamodule = Engine.from_config(CONFIG_PATH)



engine.train(
    model=model, 
    datamodule=datamodule
)

I want the get the ROC curve image after training as anomalib returns in older versions.

[samet-akcay]

Depending on your model and dataset, you should have image_ROC.png and pixel_ROC.png in the folder where images are saved. For example, here is my code;

from anomalib.data import MVTec
from anomalib.engine import Engine
from anomalib.models import Padim

datamodule = MVTec()
model = Padim()
engine = Engine()
engine.train(datamodule=datamodule, model=model)

This returns the following output:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│        image_AUROC        │    0.9992064237594604     │
│       image_F1Score       │    0.9841269850730896     │
│        pixel_AUROC        │    0.9817485809326172     │
│       pixel_F1Score       │    0.7083382606506348     │
└───────────────────────────┴───────────────────────────┘

And the ROC figures are saved into the following:

❯ tree -L 1 results/Padim/MVTec/bottle/latest/images/
results/Padim/MVTec/bottle/latest/images/
├── broken_large
├── broken_small
├── contamination
├── good
├── image_ROC.png *
└── pixel_ROC.png *

Image ROC

Pixel ROC

Note that you need to enable MetricsVisualizer() as I suggested above.

[samet-akcay]

Like I said, we will make the visualizer customizable in the next version, so you will not need to update the source code

[lilruwu]

Yes! They are there! Thank you so much I didn't realized!