[Feature] Dedicated MMClsWandbHook for MMClassification (Weights and Biases Integration)

[ayulockin]

I have raised a PR in MMDetection contributing a dedicated MMDetWandbHook for MMDetection. I was playing around with MMClassification and realized that MMClassification can use a similar dedicated hook (with few minor modifications).

Motivation

The goal of this PR is to contribute a dedicated Weights and Biases hook for MMClassification.

Modification

The PR adds two new files:

• wandblogger_hook.py where all the Weights and Biases related logic lives and,
• eval_hooks.py was added so that the MMClsWandbHook can reuse the validation results.

The feature can easily be used this:

log_config = dict(
            interval=10,
            hooks=[
                dict(type='MMClsWandbHook',
                     wandb_init_kwargs={
                         'entity': WANDB_ENTITY,
                         'project': WANDB_PROJECT_NAME
                     },
                     log_checkpoint=True,
                     log_checkpoint_metadata=True,
                     num_eval_images=100)
            ])

Use cases (Optional)

Here are some of the use cases that this PR introduces and should be helpful to the community in general.

Metrics

• The WandbLogger will automatically log training and validation metrics.
• It will log system (CPU/GPU) metrics.

Screen.Recording.2022-04-05.at.7.55.19.PM.mov

Checkpointing with Metadata

If log_checkpoint is True, the checkpoint saved at every checkpoint interval will be saved as W&B Artifacts. On top of this, if log_checkpoint_metadata is True, every checkpoint artifact will have metadata associated with it as shown in the recording below.

Screen.Recording.2022-04-05.at.7.58.24.PM.mov

Log Model Prediction 🎉

If num_eval_images > 0, at every evaluation interval, theMMClsWandbHook logs the model prediction as interactive W&B Tables. To know more about W&B Tables, please refer to the docs here. The 'MMClsWandbHook' logs the predicted class labels along with ground truth labels.

Screen.Recording.2022-04-05.at.8.03.59.PM.mov

Checklist

Before PR:

• Pre-commit or other linting tools are used to fix the potential lint issues.
• Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
• The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.
• CLA has been signed and all committers have signed the CLA in this PR.

[CLAassistant]

All committers have signed the CLA.

[codecov]

Codecov Report

Merging #764 (1f26f10) into dev (7c5ddb1) will decrease coverage by 1.34%.
The diff coverage is 22.09%.

@@            Coverage Diff             @@
##              dev     #764      +/-   ##
==========================================
- Coverage   86.80%   85.46%   -1.35%     
==========================================
  Files         130      132       +2     
  Lines        8571     8750     +179     
  Branches     1478     1513      +35     
==========================================
+ Hits         7440     7478      +38     
- Misses        909     1050     +141     
  Partials      222      222              

Flag	Coverage Δ
unittests	85.38% <22.09%> (-1.34%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmcls/core/hook/wandblogger_hook.py	18.57% <18.57%> (ø)
mmcls/core/evaluation/eval_hooks.py	28.94% <28.94%> (ø)
mmcls/apis/train.py	14.94% <100.00%> (-0.97%)	⬇️
mmcls/core/evaluation/__init__.py	100.00% <100.00%> (ø)
mmcls/core/hook/__init__.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7c5ddb1...1f26f10. Read the comment docs.

[mzr1996]

Looks good, but I haven't used Wandb, I will try it recently. 👍

[ayulockin]

Looks good, but I haven't used Wandb, I will try it recently. 👍

Looking forward to your feedback.

[ayulockin]

Hey @mzr1996, did you get a chance to play with W&B. Would love your feedback on this work?

[ayulockin]

Hey @mzr1996, sorry to bug you again but would love it if you can give some feedback on the hook that I have implemented. I think it will add value to the community. :)

[mzr1996]

Hey @mzr1996, sorry to bug you again but would love it if you can give some feedback on the hook that I have implemented. I think it will add value to the community. :)

Sorry for the late reply. This function is planned to be merged in the version v0.23.1, and I'll review it today.

[ayulockin]

Thanks for the reply. Looking forward to the review.

[mzr1996]

Hello, I have reviewed your PR and made some modifications. You can check the difference here
If you think my PR works well, please merge that PR, and then this PR will be updated.

[ayulockin]

Thanks for looking into it in such detail. I will test out the PR and let you know. :)

[ayulockin]

Hey @mzr1996 I have tested the changes and made a few of my own (they are minor). Can you please take a look and let me know if anything else is needed. Thank you again for a really helpful PR.

[ayulockin]

Hey @mzr1996, would love to get a confirmation about the changes made to the PR after your feedback. Also can you let me know what else is required from my end? Thanks. :)

[ayulockin]

Hey @mzr1996, apologises for the ping again. But I would like to know if anything else is needed for this PR?

[mzr1996]

Looks good

[lsch0lz]

Thanks for implementing this hook! I just tried it out, but can't quite get it to run. How do I log into my WandDB Account or do I have to specify somewhere an API Key to log the informations into my WanDB Project? Training works as usual, but I cant see the logs in the specified WanDB Project.

My config file looks like this:

_base_ = [
    '../_base_/models/swin_transformer/tiny_224.py',
    '../_base_/datasets/imagenet_bs64_swin_224.py',
    '../_base_/schedules/imagenet_bs1024_adamw_swin.py',
    '../_base_/default_runtime.py'
]

# schedule settings
optim_wrapper = dict(clip_grad=dict(max_norm=5.0))

# Dataset
data_root = 'XXX'
train_dataloader = dict(
    dataset=dict(
        type='CustomDataset',
        data_root=data_root,
        ann_file='',
        data_prefix='train',
    ))
val_dataloader = dict(
    dataset=dict(
        type='CustomDataset',
        data_root=data_root,
        ann_file='',
        data_prefix='val',
    ))
test_dataloader = dict(
    dataset=dict(
        type='CustomDataset',
        data_root=data_root,
        ann_file='',
        data_prefix='test',
    ))

# Logging
log_config = dict(
            interval=10,
            hooks=[
                dict(type='MMClsWandbHook',
                     wandb_init_kwargs={
                         'entity': "MY_USER_NAME",
                         'project': "MY_PROJECT_NAME"
                     },
                     log_checkpoint=True,
                     log_checkpoint_metadata=True,
                     num_eval_images=100)
            ])