[Enhancement] Add Dev tools to boost develop #798

clownrat6 · 2021-08-17T08:22:14Z

Add more dev tools to boost develop progress (Mainly for regression benchmark)

Goals:

url check to find out invalid checkpoint url (log json url is contained but it's hacky.).
more convenient model gather tool.
test model inference of important model list.
test model evaluation of important model list.
test model training of important model list.

TODO:

url check tool: .dev/check_urls.py;
more convenient model gather tool: .dev/gather_models.py;
model-ckpt list and model_config list: .dev/batch_test_list.py & batch_train_list.txt;
model inference test tool: .dev/benchmark_inference.py;
model evaluation test tool script and script generator: .dev/benchmark_evaluation.sh & generate_benchmark_evaluation_script.py;
gather evaluation results and compare evaluation results tool: .dev/gather_benchmark_evaluation_results.py;
model training test tool scipt and script generator: .dev/benchmark_train.sh & generate_benchmark_train_script.py;
gather training results and compare training results tool: .dev/gather_benchmark_train_results.py;

codecov · 2021-08-17T08:48:27Z

Codecov Report

Merging #798 (fcbd1f1) into master (2825efe) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #798   +/-   ##
=======================================
  Coverage   88.90%   88.90%           
=======================================
  Files         110      110           
  Lines        5928     5928           
  Branches      950      950           
=======================================
  Hits         5270     5270           
  Misses        465      465           
  Partials      193      193

Flag	Coverage Δ
unittests	`88.88% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmseg/models/backbones/swin.py	`87.41% <0.00%> (ø)`
mmseg/models/backbones/resnet.py	`99.28% <0.00%> (ø)`
mmseg/models/backbones/resnest.py	`83.72% <0.00%> (ø)`
mmseg/models/backbones/resnext.py	`100.00% <0.00%> (ø)`
mmseg/models/backbones/fast_scnn.py	`97.08% <0.00%> (ø)`
mmseg/models/necks/multilevel_neck.py	`100.00% <0.00%> (ø)`
mmseg/models/backbones/mobilenet_v2.py	`71.08% <0.00%> (ø)`
mmseg/models/decode_heads/point_head.py	`96.52% <0.00%> (ø)`
mmseg/models/decode_heads/sep_fcn_head.py	`100.00% <0.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2825efe...fcbd1f1. Read the comment docs.

clownrat6 · 2021-08-19T06:28:54Z

A glance of benchmark model lists

.dev/batch_test_list.py:

hrnet = [
    dict(
        config='configs/hrnet/fcn_hr18s_512x512_160k_ade20k.py',
        checkpoint='fcn_hr18s_512x512_160k_ade20k_20200614_214413-870f65ac.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=33.0),
    ),
    dict(
        config='configs/hrnet/fcn_hr18s_512x1024_160k_cityscapes.py',
        checkpoint='fcn_hr18s_512x1024_160k_cityscapes_20200602_190901-4a0797ea.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=76.31),
    ),
    dict(
        config='configs/hrnet/fcn_hr48_512x512_160k_ade20k.py',
        checkpoint='fcn_hr48_512x512_160k_ade20k_20200614_214407-a52fc02c.pth',
        eval='mIoU',
        metric=dict(mIoU=42.02),
    ),
    dict(
        config='configs/hrnet/fcn_hr48_512x1024_160k_cityscapes.py',
        checkpoint='fcn_hr48_512x1024_160k_cityscapes_20200602_190946-59b7973e.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=80.65),
    ),
]
pspnet = [
    dict(
        config='configs/pspnet/pspnet_r50-d8_512x1024_80k_cityscapes.py',
        checkpoint='pspnet_r50-d8_512x1024_80k_cityscapes_20200606_112131-2376f12b.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=78.55),
    ),
    dict(
        config='configs/pspnet/pspnet_r101-d8_512x1024_80k_cityscapes.py',
        checkpoint='pspnet_r101-d8_512x1024_80k_cityscapes_20200606_112211-e1e1100f.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=79.76),
    ),
    dict(
        config='configs/pspnet/pspnet_r101-d8_512x512_160k_ade20k.py',
        checkpoint='pspnet_r101-d8_512x512_160k_ade20k_20200615_100650-967c316f.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=44.39),
    ),
    dict(
        config='configs/pspnet/pspnet_r50-d8_512x512_160k_ade20k.py',
        checkpoint='pspnet_r50-d8_512x512_160k_ade20k_20200615_184358-1890b0bd.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=42.48),
    ),
]
resnest = [
    dict(
        config='configs/resnest/pspnet_s101-d8_512x512_160k_ade20k.py',
        checkpoint='pspnet_s101-d8_512x512_160k_ade20k_20200807_145416-a6daa92a.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=45.44),
    ),
    dict(
        config='configs/resnest/pspnet_s101-d8_512x1024_80k_cityscapes.py',
        checkpoint='pspnet_s101-d8_512x1024_80k_cityscapes_20200807_140631-c75f3b99.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=78.57),
    ),
]
fastscnn = [
    dict(
        config='configs/fastscnn/fast_scnn_lr0.12_8x4_160k_cityscapes.py',
        checkpoint='fast_scnn_8x4_160k_lr0.12_cityscapes-0cec9937.pth',
        eval='mIoU',
        metric=dict(mIoU=70.96),
    )
]
deeplabv3plus = [
    dict(
        config='configs/deeplabv3plus/deeplabv3plus_r101-d8_769x769_80k_cityscapes.py', # noqa
        checkpoint='deeplabv3plus_r101-d8_769x769_80k_cityscapes_20200607_000405-a7573d20.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=80.98),
    ),
    dict(
        config='configs/deeplabv3plus/deeplabv3plus_r101-d8_512x1024_80k_cityscapes.py', # noqa
        checkpoint='deeplabv3plus_r101-d8_512x1024_80k_cityscapes_20200606_114143-068fcfe9.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=80.97),
    ),
    dict(
        config='configs/deeplabv3plus/deeplabv3plus_r50-d8_512x1024_80k_cityscapes.py', # noqa
        checkpoint='deeplabv3plus_r50-d8_512x1024_80k_cityscapes_20200606_114049-f9fb496d.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=80.09),
    ),
    dict(
        config='configs/deeplabv3plus/deeplabv3plus_r50-d8_769x769_80k_cityscapes.py', # noqa
        checkpoint='deeplabv3plus_r50-d8_769x769_80k_cityscapes_20200606_210233-0e9dfdc4.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=79.83),
    ),
]
vit = [
    dict(
        config='configs/vit/upernet_vit-b16_ln_mln_512x512_160k_ade20k.py',
        checkpoint='upernet_vit-b16_ln_mln_512x512_160k_ade20k-f444c077.pth',
        eval='mIoU',
        metric=dict(mIoU=47.73),
    ),
    dict(
        config='configs/vit/upernet_deit-s16_ln_mln_512x512_160k_ade20k.py',
        checkpoint='upernet_deit-s16_ln_mln_512x512_160k_ade20k-c0cd652f.pth',
        eval='mIoU',
        metric=dict(mIoU=43.52),
    ),
]
fp16 = [
    dict(
        config='configs/fp16/deeplabv3plus_r101-d8_512x1024_80k_fp16_cityscapes.py', # noqa
        checkpoint='deeplabv3plus_r101-d8_512x1024_80k_fp16_cityscapes-cc58bc8d.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=80.46),
    )
]
swin = [
    dict(
        config='configs/swin/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K.py', # noqa
        checkpoint='upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K_20210531_112542-e380ad3e.pth', # noqa
        eval='mIoU',
        metric=dict(mIoU=44.41),
    )
]

…v_tools

clownrat6 · 2021-08-24T03:25:45Z

Regression Benchmark Evaluation Results:
fcn_hr18s_512x512_160k_ade20k & fast_scnn_lr0.12_8x4_160k_cityscapes missing

{
    "configs/hrnet/fcn_hr18s_512x1024_160k_cityscapes.py": {
        "differential": {
            "mIoU": "+0.02"
        },
        "previous": {
            "mIoU": 76.31
        },
        "new": {
            "mIoU": 76.33
        }
    },
    "configs/hrnet/fcn_hr48_512x512_160k_ade20k.py": {
        "differential": {
            "mIoU": "-0.00"
        },
        "previous": {
            "mIoU": 42.02
        },
        "new": {
            "mIoU": 42.02
        }
    },
    "configs/hrnet/fcn_hr48_512x1024_160k_cityscapes.py": {
        "differential": {
            "mIoU": "+0.02"
        },
        "previous": {
            "mIoU": 80.65
        },
        "new": {
            "mIoU": 80.67
        }
    },
    "configs/pspnet/pspnet_r50-d8_512x1024_80k_cityscapes.py": {
        "differential": {
            "mIoU": "+0.11"
        },
        "previous": {
            "mIoU": 78.55
        },
        "new": {
            "mIoU": 78.66
        }
    },
    "configs/pspnet/pspnet_r101-d8_512x1024_80k_cityscapes.py": {
        "differential": {
            "mIoU": "-0.02"
        },
        "previous": {
            "mIoU": 79.76
        },
        "new": {
            "mIoU": 79.74
        }
    },
    "configs/pspnet/pspnet_r101-d8_512x512_160k_ade20k.py": {
        "differential": {
            "mIoU": "-0.00"
        },
        "previous": {
            "mIoU": 44.39
        },
        "new": {
            "mIoU": 44.39
        }
    },
    "configs/pspnet/pspnet_r50-d8_512x512_160k_ade20k.py": {
        "differential": {
            "mIoU": "-0.00"
        },
        "previous": {
            "mIoU": 42.48
        },
        "new": {
            "mIoU": 42.48
        }
    },
    "configs/resnest/pspnet_s101-d8_512x512_160k_ade20k.py": {
        "differential": {
            "mIoU": "-0.00"
        },
        "previous": {
            "mIoU": 45.44
        },
        "new": {
            "mIoU": 45.44
        }
    },
    "configs/resnest/pspnet_s101-d8_512x1024_80k_cityscapes.py": {
        "differential": {
            "mIoU": "-0.00"
        },
        "previous": {
            "mIoU": 78.57
        },
        "new": {
            "mIoU": 78.57
        }
    },
    "configs/deeplabv3plus/deeplabv3plus_r101-d8_769x769_80k_cityscapes.py": {
        "differential": {
            "mIoU": "-0.06"
        },
        "previous": {
            "mIoU": 80.98
        },
        "new": {
            "mIoU": 80.92
        }
    },
    "configs/deeplabv3plus/deeplabv3plus_r101-d8_512x1024_80k_cityscapes.py": {
        "differential": {
            "mIoU": "+0.07"
        },
        "previous": {
            "mIoU": 80.97
        },
        "new": {
            "mIoU": 81.04
        }
    },
    "configs/deeplabv3plus/deeplabv3plus_r50-d8_512x1024_80k_cityscapes.py": {
        "differential": {
            "mIoU": "-0.08"
        },
        "previous": {
            "mIoU": 80.09
        },
        "new": {
            "mIoU": 80.01
        }
    },
    "configs/deeplabv3plus/deeplabv3plus_r50-d8_769x769_80k_cityscapes.py": {
        "differential": {
            "mIoU": "-0.00"
        },
        "previous": {
            "mIoU": 79.83
        },
        "new": {
            "mIoU": 79.83
        }
    },
    "configs/vit/upernet_vit-b16_ln_mln_512x512_160k_ade20k.py": {
        "differential": {
            "mIoU": "+1.04"
        },
        "previous": {
            "mIoU": 47.73
        },
        "new": {
            "mIoU": 48.77
        }
    },
    "configs/vit/upernet_deit-s16_ln_mln_512x512_160k_ade20k.py": {
        "differential": {
            "mIoU": "+0.67"
        },
        "previous": {
            "mIoU": 43.52
        },
        "new": {
            "mIoU": 44.19
        }
    },
    "configs/fp16/deeplabv3plus_r101-d8_512x1024_80k_fp16_cityscapes.py": {
        "differential": {
            "mIoU": "-0.00"
        },
        "previous": {
            "mIoU": 80.46
        },
        "new": {
            "mIoU": 80.46
        }
    },
    "configs/swin/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K.py": {
        "differential": {
            "mIoU": "-0.00"
        },
        "previous": {
            "mIoU": 44.41
        },
        "new": {
            "mIoU": 44.41
        }
    }
}

clownrat6 · 2021-08-24T03:26:18Z

Regression Benchmark Training Results:

{
    "configs/hrnet/fcn_hr18s_512x512_160k_ade20k.py": {
        "mIoU": 32.9,
        "mAcc": 43.1,
        "aAcc": 76.9
    },
    "configs/hrnet/fcn_hr18s_512x1024_160k_cityscapes.py": {
        "mIoU": 76.6,
        "mAcc": 84.5,
        "aAcc": 96.0
    },
    "configs/hrnet/fcn_hr48_512x512_160k_ade20k.py": {
        "mIoU": 42.0,
        "mAcc": 53.6,
        "aAcc": 80.2
    },
    "configs/hrnet/fcn_hr48_512x1024_160k_cityscapes.py": {
        "mIoU": 81.0,
        "mAcc": 87.9,
        "aAcc": 96.7
    },
    "configs/pspnet/pspnet_r50-d8_512x1024_80k_cityscapes.py": {
        "mIoU": 78.4,
        "mAcc": 85.2,
        "aAcc": 96.2
    },
    "configs/pspnet/pspnet_r101-d8_512x1024_80k_cityscapes.py": {
        "mIoU": 80.1,
        "mAcc": 87.1,
        "aAcc": 96.4
    },
    "configs/pspnet/pspnet_r101-d8_512x512_160k_ade20k.py": {
        "mIoU": 43.8,
        "mAcc": 53.8,
        "aAcc": 81.2
    },
    "configs/pspnet/pspnet_r50-d8_512x512_160k_ade20k.py": {
        "mIoU": 42.4,
        "mAcc": 52.2,
        "aAcc": 80.6
    },
    "configs/resnest/pspnet_s101-d8_512x512_160k_ade20k.py": {
        "mIoU": 44.6,
        "mAcc": 55.8,
        "aAcc": 81.2
    },
    "configs/fastscnn/fast_scnn_lr0.12_8x4_160k_cityscapes.py": {
        "mIoU": 71.0,
        "mAcc": 78.8,
        "aAcc": 94.9
    },
    "configs/deeplabv3plus/deeplabv3plus_r101-d8_769x769_80k_cityscapes.py": {
        "mIoU": 81.0,
        "mAcc": 88.8,
        "aAcc": 96.5
    },
    "configs/deeplabv3plus/deeplabv3plus_r101-d8_512x1024_80k_cityscapes.py": {
        "mIoU": 81.1,
        "mAcc": 87.8,
        "aAcc": 96.6
    },
    "configs/deeplabv3plus/deeplabv3plus_r50-d8_512x1024_80k_cityscapes.py": {
        "mIoU": 80.6,
        "mAcc": 87.9,
        "aAcc": 96.5
    },
    "configs/deeplabv3plus/deeplabv3plus_r50-d8_769x769_80k_cityscapes.py": {
        "mIoU": 80.1,
        "mAcc": 88.4,
        "aAcc": 96.5
    },
    "configs/vit/upernet_vit-b16_ln_mln_512x512_160k_ade20k.py": {
        "mIoU": 48.1,
        "mAcc": 58.9,
        "aAcc": 82.6
    },
    "configs/vit/upernet_deit-s16_ln_mln_512x512_160k_ade20k.py": {
        "mIoU": 44.2,
        "mAcc": 55.2,
        "aAcc": 81.1
    },
    "configs/fp16/deeplabv3plus_r101-d8_512x1024_80k_fp16_cityscapes.py": {
        "mIoU": 80.7,
        "mAcc": 87.4,
        "aAcc": 96.5
    },
    "configs/swin/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K.py": {
        "mIoU": 44.0,
        "mAcc": 55.0,
        "aAcc": 81.0
    }
}

Junjun2016 · 2021-08-24T05:10:30Z

Regression Benchmark Training Results:

{
    "configs/hrnet/fcn_hr18s_512x512_160k_ade20k.py": {
        "mIoU": 32.9,
        "mAcc": 43.1,
        "aAcc": 76.9
    },
    "configs/hrnet/fcn_hr18s_512x1024_160k_cityscapes.py": {
        "mIoU": 76.6,
        "mAcc": 84.5,
        "aAcc": 96.0
    },
    "configs/hrnet/fcn_hr48_512x512_160k_ade20k.py": {
        "mIoU": 42.0,
        "mAcc": 53.6,
        "aAcc": 80.2
    },
    "configs/hrnet/fcn_hr48_512x1024_160k_cityscapes.py": {
        "mIoU": 81.0,
        "mAcc": 87.9,
        "aAcc": 96.7
    },
    "configs/pspnet/pspnet_r50-d8_512x1024_80k_cityscapes.py": {
        "mIoU": 78.4,
        "mAcc": 85.2,
        "aAcc": 96.2
    },
    "configs/pspnet/pspnet_r101-d8_512x1024_80k_cityscapes.py": {
        "mIoU": 80.1,
        "mAcc": 87.1,
        "aAcc": 96.4
    },
    "configs/pspnet/pspnet_r101-d8_512x512_160k_ade20k.py": {
        "mIoU": 43.8,
        "mAcc": 53.8,
        "aAcc": 81.2
    },
    "configs/pspnet/pspnet_r50-d8_512x512_160k_ade20k.py": {
        "mIoU": 42.4,
        "mAcc": 52.2,
        "aAcc": 80.6
    },
    "configs/resnest/pspnet_s101-d8_512x512_160k_ade20k.py": {
        "mIoU": 44.6,
        "mAcc": 55.8,
        "aAcc": 81.2
    },
    "configs/fastscnn/fast_scnn_lr0.12_8x4_160k_cityscapes.py": {
        "mIoU": 71.0,
        "mAcc": 78.8,
        "aAcc": 94.9
    },
    "configs/deeplabv3plus/deeplabv3plus_r101-d8_769x769_80k_cityscapes.py": {
        "mIoU": 81.0,
        "mAcc": 88.8,
        "aAcc": 96.5
    },
    "configs/deeplabv3plus/deeplabv3plus_r101-d8_512x1024_80k_cityscapes.py": {
        "mIoU": 81.1,
        "mAcc": 87.8,
        "aAcc": 96.6
    },
    "configs/deeplabv3plus/deeplabv3plus_r50-d8_512x1024_80k_cityscapes.py": {
        "mIoU": 80.6,
        "mAcc": 87.9,
        "aAcc": 96.5
    },
    "configs/deeplabv3plus/deeplabv3plus_r50-d8_769x769_80k_cityscapes.py": {
        "mIoU": 80.1,
        "mAcc": 88.4,
        "aAcc": 96.5
    },
    "configs/vit/upernet_vit-b16_ln_mln_512x512_160k_ade20k.py": {
        "mIoU": 48.1,
        "mAcc": 58.9,
        "aAcc": 82.6
    },
    "configs/vit/upernet_deit-s16_ln_mln_512x512_160k_ade20k.py": {
        "mIoU": 44.2,
        "mAcc": 55.2,
        "aAcc": 81.1
    },
    "configs/fp16/deeplabv3plus_r101-d8_512x1024_80k_fp16_cityscapes.py": {
        "mIoU": 80.7,
        "mAcc": 87.4,
        "aAcc": 96.5
    },
    "configs/swin/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K.py": {
        "mIoU": 44.0,
        "mAcc": 55.0,
        "aAcc": 81.0
    }
}

Are the results in this file equal to the results of new in the above file?

Junjun2016 · 2021-08-29T07:20:48Z

Hi @Sennnnn
Please fix the lint error.
update-model-index hook has been fixed.

…v_tools

Junjun2016

Please resolve comments.

.dev/check_urls.py

.dev/generate_benchmark_train_script.py

.dev/gather_benchmark_train_results.py

.dev/gather_models.py

.dev/gather_benchmark_train_results.py

…odify gather models tool.

.dev/check_urls.py

Junjun2016

LGTM

xvjiarui · 2021-08-30T18:22:48Z

tools/train.py

+        config_dir = osp.split(osp.dirname(args.config))[1]
+        config_name = osp.splitext(osp.basename(args.config))[0]
+        cfg.work_dir = osp.join('./work_dirs', config_dir, config_name)


Please take the following case into consideration:

The user added 1 config file under the folder configs/, or directory under the root of repo. config_dir will become configs or MMSegmentation, which is not idea.

I suggest keep this part unchanged.

xvjiarui · 2021-08-30T18:34:11Z

tools/test.py

@@ -21,6 +23,9 @@ def parse_args():
        description='mmseg test (and eval) a model')
    parser.add_argument('config', help='test config file path')
    parser.add_argument('checkpoint', help='checkpoint file')
+    parser.add_argument(
+        '--work-dir',
+        help='the directory to save the file containing evaluation metrics')


Suggested change

help='the directory to save the file containing evaluation metrics')

help='if specified, the evaluation metric results will be dumped into the directory as json')

xvjiarui · 2021-09-01T02:17:47Z

.dev/benchmark_evaluation.sh

+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION fcn_hr18s_512x512_160k_ade20k configs/hrnet/fcn_hr18s_512x512_160k_ade20k.py $CHECKPOINT_DIR/fcn_hr18s_512x512_160k_ade20k_20200614_214413-870f65ac.pth --eval mIoU --work-dir .dev/benchmark_test/fcn_hr18s_512x512_160k_ade20k --options dist_params.port=28171 &
+echo 'configs/hrnet/fcn_hr18s_512x1024_160k_cityscapes.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION fcn_hr18s_512x1024_160k_cityscapes configs/hrnet/fcn_hr18s_512x1024_160k_cityscapes.py $CHECKPOINT_DIR/fcn_hr18s_512x1024_160k_cityscapes_20200602_190901-4a0797ea.pth --eval mIoU --work-dir .dev/benchmark_test/fcn_hr18s_512x1024_160k_cityscapes --options dist_params.port=28172 &
+echo 'configs/hrnet/fcn_hr48_512x512_160k_ade20k.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION fcn_hr48_512x512_160k_ade20k configs/hrnet/fcn_hr48_512x512_160k_ade20k.py $CHECKPOINT_DIR/fcn_hr48_512x512_160k_ade20k_20200614_214407-a52fc02c.pth --eval mIoU --work-dir .dev/benchmark_test/fcn_hr48_512x512_160k_ade20k --options dist_params.port=28173 &
+echo 'configs/hrnet/fcn_hr48_512x1024_160k_cityscapes.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION fcn_hr48_512x1024_160k_cityscapes configs/hrnet/fcn_hr48_512x1024_160k_cityscapes.py $CHECKPOINT_DIR/fcn_hr48_512x1024_160k_cityscapes_20200602_190946-59b7973e.pth --eval mIoU --work-dir .dev/benchmark_test/fcn_hr48_512x1024_160k_cityscapes --options dist_params.port=28174 &
+echo 'configs/pspnet/pspnet_r50-d8_512x1024_80k_cityscapes.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION pspnet_r50-d8_512x1024_80k_cityscapes configs/pspnet/pspnet_r50-d8_512x1024_80k_cityscapes.py $CHECKPOINT_DIR/pspnet_r50-d8_512x1024_80k_cityscapes_20200606_112131-2376f12b.pth --eval mIoU --work-dir .dev/benchmark_test/pspnet_r50-d8_512x1024_80k_cityscapes --options dist_params.port=28175 &
+echo 'configs/pspnet/pspnet_r101-d8_512x1024_80k_cityscapes.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION pspnet_r101-d8_512x1024_80k_cityscapes configs/pspnet/pspnet_r101-d8_512x1024_80k_cityscapes.py $CHECKPOINT_DIR/pspnet_r101-d8_512x1024_80k_cityscapes_20200606_112211-e1e1100f.pth --eval mIoU --work-dir .dev/benchmark_test/pspnet_r101-d8_512x1024_80k_cityscapes --options dist_params.port=28176 &
+echo 'configs/pspnet/pspnet_r101-d8_512x512_160k_ade20k.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION pspnet_r101-d8_512x512_160k_ade20k configs/pspnet/pspnet_r101-d8_512x512_160k_ade20k.py $CHECKPOINT_DIR/pspnet_r101-d8_512x512_160k_ade20k_20200615_100650-967c316f.pth --eval mIoU --work-dir .dev/benchmark_test/pspnet_r101-d8_512x512_160k_ade20k --options dist_params.port=28177 &
+echo 'configs/pspnet/pspnet_r50-d8_512x512_160k_ade20k.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION pspnet_r50-d8_512x512_160k_ade20k configs/pspnet/pspnet_r50-d8_512x512_160k_ade20k.py $CHECKPOINT_DIR/pspnet_r50-d8_512x512_160k_ade20k_20200615_184358-1890b0bd.pth --eval mIoU --work-dir .dev/benchmark_test/pspnet_r50-d8_512x512_160k_ade20k --options dist_params.port=28178 &
+echo 'configs/resnest/pspnet_s101-d8_512x512_160k_ade20k.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION pspnet_s101-d8_512x512_160k_ade20k configs/resnest/pspnet_s101-d8_512x512_160k_ade20k.py $CHECKPOINT_DIR/pspnet_s101-d8_512x512_160k_ade20k_20200807_145416-a6daa92a.pth --eval mIoU --work-dir .dev/benchmark_test/pspnet_s101-d8_512x512_160k_ade20k --options dist_params.port=28179 &
+echo 'configs/resnest/pspnet_s101-d8_512x1024_80k_cityscapes.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION pspnet_s101-d8_512x1024_80k_cityscapes configs/resnest/pspnet_s101-d8_512x1024_80k_cityscapes.py $CHECKPOINT_DIR/pspnet_s101-d8_512x1024_80k_cityscapes_20200807_140631-c75f3b99.pth --eval mIoU --work-dir .dev/benchmark_test/pspnet_s101-d8_512x1024_80k_cityscapes --options dist_params.port=28180 &
+echo 'configs/fastscnn/fast_scnn_lr0.12_8x4_160k_cityscapes.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION fast_scnn_lr0.12_8x4_160k_cityscapes configs/fastscnn/fast_scnn_lr0.12_8x4_160k_cityscapes.py $CHECKPOINT_DIR/fast_scnn_8x4_160k_lr0.12_cityscapes-0cec9937.pth --eval mIoU --work-dir .dev/benchmark_test/fast_scnn_lr0.12_8x4_160k_cityscapes --options dist_params.port=28181 &
+echo 'configs/deeplabv3plus/deeplabv3plus_r101-d8_769x769_80k_cityscapes.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION deeplabv3plus_r101-d8_769x769_80k_cityscapes configs/deeplabv3plus/deeplabv3plus_r101-d8_769x769_80k_cityscapes.py $CHECKPOINT_DIR/deeplabv3plus_r101-d8_769x769_80k_cityscapes_20200607_000405-a7573d20.pth --eval mIoU --work-dir .dev/benchmark_test/deeplabv3plus_r101-d8_769x769_80k_cityscapes --options dist_params.port=28182 &
+echo 'configs/deeplabv3plus/deeplabv3plus_r101-d8_512x1024_80k_cityscapes.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION deeplabv3plus_r101-d8_512x1024_80k_cityscapes configs/deeplabv3plus/deeplabv3plus_r101-d8_512x1024_80k_cityscapes.py $CHECKPOINT_DIR/deeplabv3plus_r101-d8_512x1024_80k_cityscapes_20200606_114143-068fcfe9.pth --eval mIoU --work-dir .dev/benchmark_test/deeplabv3plus_r101-d8_512x1024_80k_cityscapes --options dist_params.port=28183 &
+echo 'configs/deeplabv3plus/deeplabv3plus_r50-d8_512x1024_80k_cityscapes.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION deeplabv3plus_r50-d8_512x1024_80k_cityscapes configs/deeplabv3plus/deeplabv3plus_r50-d8_512x1024_80k_cityscapes.py $CHECKPOINT_DIR/deeplabv3plus_r50-d8_512x1024_80k_cityscapes_20200606_114049-f9fb496d.pth --eval mIoU --work-dir .dev/benchmark_test/deeplabv3plus_r50-d8_512x1024_80k_cityscapes --options dist_params.port=28184 &
+echo 'configs/deeplabv3plus/deeplabv3plus_r50-d8_769x769_80k_cityscapes.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION deeplabv3plus_r50-d8_769x769_80k_cityscapes configs/deeplabv3plus/deeplabv3plus_r50-d8_769x769_80k_cityscapes.py $CHECKPOINT_DIR/deeplabv3plus_r50-d8_769x769_80k_cityscapes_20200606_210233-0e9dfdc4.pth --eval mIoU --work-dir .dev/benchmark_test/deeplabv3plus_r50-d8_769x769_80k_cityscapes --options dist_params.port=28185 &
+echo 'configs/vit/upernet_vit-b16_ln_mln_512x512_160k_ade20k.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION upernet_vit-b16_ln_mln_512x512_160k_ade20k configs/vit/upernet_vit-b16_ln_mln_512x512_160k_ade20k.py $CHECKPOINT_DIR/upernet_vit-b16_ln_mln_512x512_160k_ade20k-f444c077.pth --eval mIoU --work-dir .dev/benchmark_test/upernet_vit-b16_ln_mln_512x512_160k_ade20k --options dist_params.port=28186 &
+echo 'configs/vit/upernet_deit-s16_ln_mln_512x512_160k_ade20k.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION upernet_deit-s16_ln_mln_512x512_160k_ade20k configs/vit/upernet_deit-s16_ln_mln_512x512_160k_ade20k.py $CHECKPOINT_DIR/upernet_deit-s16_ln_mln_512x512_160k_ade20k-c0cd652f.pth --eval mIoU --work-dir .dev/benchmark_test/upernet_deit-s16_ln_mln_512x512_160k_ade20k --options dist_params.port=28187 &
+echo 'configs/fp16/deeplabv3plus_r101-d8_512x1024_80k_fp16_cityscapes.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION deeplabv3plus_r101-d8_512x1024_80k_fp16_cityscapes configs/fp16/deeplabv3plus_r101-d8_512x1024_80k_fp16_cityscapes.py $CHECKPOINT_DIR/deeplabv3plus_r101-d8_512x1024_80k_fp16_cityscapes-cc58bc8d.pth --eval mIoU --work-dir .dev/benchmark_test/deeplabv3plus_r101-d8_512x1024_80k_fp16_cityscapes --options dist_params.port=28188 &
+echo 'configs/swin/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K.py' &
+GPUS=4  GPUS_PER_NODE=4  CPUS_PER_TASK=2 tools/slurm_test.sh $PARTITION upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K configs/swin/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K.py $CHECKPOINT_DIR/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K_20210531_112542-e380ad3e.pth --eval mIoU --work-dir .dev/benchmark_test/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K --options dist_params.port=28189 &


We may specify work_dir for these scripts.

Junjun2016

gather_model should also be modified.

Junjun2016 · 2021-09-02T07:00:08Z

\ --> /

Junjun2016

LGTM

* Modify default work dir when training. * Refactor gather_models.py. * Add train and test matching list. * Regression benchmark list. * lower readme name to upper readme name. * Add url check tool and model inference test tool. * Modify tool name. * Support duplicate mode of log json url check. * Add regression benchmark evaluation automatic tool. * Add train script generator. * Only Support script running. * Add evaluation results gather. * Add exec Authority. * Automatically make checkpoint root folder. * Modify gather results save path. * Coarse-grained train results gather tool. * Complete benchmark train script. * Make some little modifications. * Fix checkpoint urls. * Fix unet checkpoint urls. * Fix fast scnn & fcn checkpoint url. * Fix fast scnn checkpoint urls. * Fix fast scnn url. * Add differential results calculation. * Add differential results of regression benchmark train results. * Add an extra argument to select model. * Update nonlocal_net & hrnet checkpoint url. * Fix checkpoint url of hrnet and Fix some tta evaluation results and modify gather models tool. * Modify fast scnn checkpoint url. * Resolve new comments. * Fix url check status code bug. * Resolve some comments. * Modify train scripts generator. * Modify work_dir of regression benchmark results. * model gather tool modification.

fix typo docstring

* update install * resolve comments * rm mim * resolve comments

* Add ref to mmocr * Add chinese version

sennnnn added 9 commits August 12, 2021 18:12

Modify default work dir when training.

9895e34

Refactor gather_models.py.

3b2b8c5

Add train and test matching list.

37a1872

Regression benchmark list.

828522b

Merge branch 'master' into dev_tools

cb42556

Merge branch 'master' into dev_tools

2ffb252

lower readme name to upper readme name.

d4ff6b8

Add url check tool and model inference test tool.

104d721

Modify tool name.

5378a1d

sennnnn added 4 commits August 17, 2021 19:01

Support duplicate mode of log json url check.

f25d6a8

Merge branch 'master' into dev_tools

10822df

Merge Master.

06a06fd

Add regression benchmark evaluation automatic tool.

6bd2215

sennnnn added 11 commits August 20, 2021 15:56

Add train script generator.

b25dc95

Only Support script running.

055fecd

Add evaluation results gather.

bad757b

Add exec Authority.

e39f86d

Automatically make checkpoint root folder.

fbee304

Merge branch 'dev_tools' of github.com:sennnnn/mmsegmentation into de…

84b2461

…v_tools

Modify gather results save path.

f1973eb

Coarse-grained train results gather tool.

9eee96b

Complete benchmark train script.

57439c0

Make some little modifications.

665c761

Merge Master.

44e58a8

clownrat6 requested a review from Junjun2016 August 21, 2021 09:31

Add differential results of regression benchmark train results.

a50f749

sennnnn added 3 commits August 29, 2021 17:26

Add an extra argument to select model.

b524142

Merge branch 'dev_tools' of github.com:sennnnn/mmsegmentation into de…

ad022b5

…v_tools

Update nonlocal_net & hrnet checkpoint url.

2cf1bea

Junjun2016 reviewed Aug 30, 2021

View reviewed changes

sennnnn added 4 commits August 30, 2021 22:59

Fix checkpoint url of hrnet and Fix some tta evaluation results and m…

97dc719

…odify gather models tool.

Modify fast scnn checkpoint url.

27866dc

Merge Master.

7787b8e

Resolve new comments.

4f93fec

Junjun2016 reviewed Aug 30, 2021

View reviewed changes

.dev/check_urls.py Outdated Show resolved Hide resolved

Fix url check status code bug.

71a65ec

Junjun2016 approved these changes Aug 30, 2021

View reviewed changes

Junjun2016 requested a review from xvjiarui August 30, 2021 18:14

xvjiarui reviewed Aug 30, 2021

View reviewed changes

Resolve some comments.

de077e1

xvjiarui reviewed Sep 1, 2021

View reviewed changes

sennnnn added 2 commits September 2, 2021 14:48

Modify train scripts generator.

b9eb913

Modify work_dir of regression benchmark results.

1ee7564

Junjun2016 reviewed Sep 2, 2021

View reviewed changes

model gather tool modification.

fcbd1f1

Junjun2016 approved these changes Sep 2, 2021

View reviewed changes

xvjiarui approved these changes Sep 2, 2021

View reviewed changes

xvjiarui merged commit d35fbbd into open-mmlab:master Sep 2, 2021

aravind-h-v pushed a commit to aravind-h-v/mmsegmentation that referenced this pull request Mar 27, 2023

fix typo docstring in unet2d (open-mmlab#798)

71ca10c

fix typo docstring

wjkim81 pushed a commit to wjkim81/mmsegmentation that referenced this pull request Dec 3, 2023

Translate install (open-mmlab#798)

a84f675

* update install * resolve comments * rm mim * resolve comments

sibozhang pushed a commit to sibozhang/mmsegmentation that referenced this pull request Mar 22, 2024

Add ref to mmocr (open-mmlab#798)

c4978a5

* Add ref to mmocr * Add chinese version

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Add Dev tools to boost develop #798

[Enhancement] Add Dev tools to boost develop #798

clownrat6 commented Aug 17, 2021 •

edited

Loading

codecov bot commented Aug 17, 2021 •

edited

Loading

clownrat6 commented Aug 19, 2021 •

edited

Loading

clownrat6 commented Aug 24, 2021

clownrat6 commented Aug 24, 2021

Junjun2016 commented Aug 24, 2021

Junjun2016 commented Aug 29, 2021

Junjun2016 left a comment

Junjun2016 left a comment

xvjiarui Aug 30, 2021

xvjiarui Aug 30, 2021

xvjiarui Aug 30, 2021

xvjiarui Sep 1, 2021

Junjun2016 left a comment

Junjun2016 commented Sep 2, 2021

Junjun2016 left a comment

	help='the directory to save the file containing evaluation metrics')
	help='if specified, the evaluation metric results will be dumped into the directory as json')

[Enhancement] Add Dev tools to boost develop #798

[Enhancement] Add Dev tools to boost develop #798

Conversation

clownrat6 commented Aug 17, 2021 • edited Loading

Add more dev tools to boost develop progress (Mainly for regression benchmark)

codecov bot commented Aug 17, 2021 • edited Loading

Codecov Report

clownrat6 commented Aug 19, 2021 • edited Loading

A glance of benchmark model lists

clownrat6 commented Aug 24, 2021

clownrat6 commented Aug 24, 2021

Junjun2016 commented Aug 24, 2021

Junjun2016 commented Aug 29, 2021

Junjun2016 left a comment

Choose a reason for hiding this comment

Junjun2016 left a comment

Choose a reason for hiding this comment

xvjiarui Aug 30, 2021

Choose a reason for hiding this comment

xvjiarui Aug 30, 2021

Choose a reason for hiding this comment

xvjiarui Aug 30, 2021

Choose a reason for hiding this comment

xvjiarui Sep 1, 2021

Choose a reason for hiding this comment

Junjun2016 left a comment

Choose a reason for hiding this comment

Junjun2016 commented Sep 2, 2021

Junjun2016 left a comment

Choose a reason for hiding this comment

clownrat6 commented Aug 17, 2021 •

edited

Loading

codecov bot commented Aug 17, 2021 •

edited

Loading

clownrat6 commented Aug 19, 2021 •

edited

Loading