Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Doc][Fix]Refine train, predict and evaluate docs for release 2.7 #2780

Merged
merged 10 commits into from
Nov 29, 2022
Merged

[Doc][Fix]Refine train, predict and evaluate docs for release 2.7 #2780

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
447549a
[Doc][Fix]Refine train, predict, evaluate docs for release 2.7
juncaipeng Nov 28, 2022
737a869
add config preparation doc
juncaipeng Nov 29, 2022
998ab74
up
juncaipeng Nov 29, 2022
6a46806
up
juncaipeng Nov 29, 2022
69ed7e8
up
juncaipeng Nov 29, 2022
22292e0
add model zoo
juncaipeng Nov 29, 2022
7454773
u
juncaipeng Nov 29, 2022
20771f8
up
juncaipeng Nov 29, 2022
b829db0
up
juncaipeng Nov 29, 2022
d1c91eb
up
juncaipeng Nov 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add model zoo
  • Loading branch information
@juncaipeng
juncaipeng committed Nov 29, 2022
commit 22292e0d4309981cdd00f6102b293d12be3070c3
1 change: 1 addition & 0 deletions README_CN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -367,6 +367,7 @@
* [安装说明](./docs/install_cn.md)
* [快速体验](./docs/quick_start_cn.md)
* [20分钟快速上手PaddleSeg](./docs/whole_process_cn.md)
* [模型库](./docs/model_zoo_overview_cn.md)

**基础教程**

Expand Down
7 changes: 4 additions & 3 deletions README_EN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -369,11 +369,12 @@ Note that:

* [Installation](./docs/install.md)
* [Quick Start](./docs/quick_start.md)
* [A 20 minutes Blitz to learn PaddleSeg](./docs/whole_process.md)
* [A 20 minutes Blitz to Learn PaddleSeg](./docs/whole_process.md)
* [Model Zoo](./docs/model_zoo_overview.md)

**Basic Tutorials**

* Data Preparation
* Data Preparation
* [Prepare Public Dataset](./docs/data/pre_data.md)
* [Prepare Customized Dataset](./docs/data/marker/marker.md)
* [Label Data with EISeg](./EISeg)
Expand All @@ -386,7 +387,7 @@ Note that:
* [Export Inference Model](./docs/model_export.md)
* [Export ONNX Model](./docs/model_export_onnx.md)

* Model Deploy
* Model Deploy
* [Paddle Inference (Python)](./docs/deployment/inference/python_inference.md)
* [Paddle Inference (C++)](./docs/deployment/inference/cpp_inference.md)
* [Paddle Lite](./docs/deployment/lite/lite.md)
Expand Down
107 changes: 19 additions & 88 deletions docs/model_zoo_overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,24 @@
English | [简体中文](model_zoo_overview_cn.md)

# PaddleSeg model zoo overview
# Model Zoo

## Model zoo
### CNN Series
## Semantic Segmentation Model Zoo

目前，PaddleSeg提供了40+语义分割模型、150+预训练模型、10+骨干网络。

在`PaddleSeg/configs`目录下，我们给出了所有模型在公开数据集上的配置文件和README文件，比如[PP-LiteSeg模型](../configs/pp_liteseg/)。
其中，README文件介绍了模型论文和训练精度，而且给出了公开数据集上训练的权重，可以直接下载进行测试。

如下我们列出了部分模型。

PaddleSeg provides 40+ semantic segmentation models, 150+ well-trained models, 10+ backbones.

In `PaddleSeg/configs`, we provide the config files and readme.md for all models on common dataset, e.g., [PP-LiteSeg](../configs/pp_liteseg/).
Besides, the readme.md file introduces the origin paper, the performance and the trained weights.

Some common models are as follows.

**CNN Series**

|Model\Backbone Network|ResNet50|ResNet101|HRNetw18|HRNetw48|
|-|-|-|-|-|
Expand Down Expand Up @@ -49,92 +64,8 @@ English | [简体中文](model_zoo_overview_cn.md)
|[GloRe](../configs/glore)|✔|-|-|-|
|[PP-LiteSeg](../configs/pp_liteseg)|-|-|-|-|

### Transformer series
**Transformer series**
* [SETR](../configs/setr)
* [MLATransformer](../contrib/AutoNUE/configs)
* [SegFormer](../configs/segformer)
* [SegMenter](../configs/segmenter)

# Model zoo benchmark
Based on the Cityscapes dataset, PaddleSeg supports 22+ series of segmentation algorithms and corresponding 30+ image segmentation pre-training models. The performance is evaluated as follows.

**Test environment:**

- GPU: Tesla V100 16GB
- CPU: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
- CUDA: 10.2
- cuDNN: 7.6
- Paddle: 2.1.3
- PaddleSeg: 2.3

**Test method:**

- Single GPU, Batch size is 1, the running time is pure model prediction time, and the predicted image size is 1024x512.
- Use Paddle Inference's Python API to test the model after export.
- Inference time is the result of averaging predictions using 100 images in the CityScapes dataset.
- Some algorithms have only tested performance under the configuration that achieves the highest segmentation accuracy.

## Accuracy vs Speed
<div align="center">
<img src=https://user-images.githubusercontent.com/30695251/140323144-c44671ac-8ff8-4c11-a6f7-1ea339c27852.png //>
</div>

## Accuracy vs FLOPs
<div align="center">
<img src=https://user-images.githubusercontent.com/30695251/140323107-02ce9de4-c8f4-4f18-88b2-59bd0055a70b.png //>
</div>

## Accuracy vs Params
<div align="center">
<img src=https://user-images.githubusercontent.com/30695251/140323131-ed03fbb1-a583-47f5-a7dd-f4ea3582c345.png //>
</div>

## Summary table
|Model|Backbone|mIoU|Flops(G)|Params(M)|Inference Time(ms)|Preprocess Time(ms)|Postprocess Time(ms)
|-|-|-|-|-|-|-|-|
|BiSeNetv2|-|73.19%|16.14|2.33|16.00|167.45|0.013
|Fast-SCNN|-|69.31%|2.04|1.44|10.43|161.52|0.012
|HarDNet|-|79.03%|35.40|4.13|21.19|164.36|0.013
|U-Net|-|65.00%|253.75|13.41|29.11|137.75|0.012
|SegFormer_B0|-|76.73%|13.63|3.72|15.66|152.60|0.017
|SegFormer_B1|-|78.35%|26.55|13.68|21.48|152.40|0.017
|STDC1-Seg50|STDC1|74.74%|24.83|8.29|9.10|153.01|0.016
|STDC2-Seg50|STDC2|77.60%|38.05|12.33|10.88|152.64|0.015
|ANN|ResNet101|79.50%|564.43|67.70|94.91|143.35|0.013
|DANet|ResNet50|80.27%|398.48|47.52|95.08|134.78|0.015
|Deeplabv3|ResNet101_OS8|80.85%|481.00|58.17|114|141.65|0.014
|Deeplabv3P|ResNet50_OS8|81.10%|228.44|26.79|69.78|147.24|0.016
|FCN|HRNet_W48|80.70%|187.50|65.94|45.46|130.58|0.012
|GCNet|ResNet101_OS8|81.01%|570.74|68.73|90.28|119.38|0.013
|OCRNet|HRNet_W48|82.15%|324.66|70.47|61.88|138.48|0.014
|PSPNet|ResNet101_OS8|80.48%|686.89|86.97|115.93|115.94|0.012
|DecoupledSegNet|ResNet50_OS8|81.26%|395.10|41.71|66.89|136.28|0.013
|EMANet|ResNet101_OS8|80.00%|512.18|61.45|80.05|140.47|0.013
|ISANet|ResNet101_OS8|80.10%|474.13|56.81|91.72|129.12|0.012
|DNLNet|ResNet101_OS8|81.03%|575.04|69.13|97.81|138.95|0.014
|SFNet|ResNet18_OS8|78.72%|136.80|13.81|69.51|131.67|0.015
|SFNet|ResNet50_OS8|81.49%|394.37|42.03|121.35|160.45|0.013
|PointRend|ResNet50_OS8|76.54%|363.17|28.18|70.35|157.24|0.016
|SegFormer_B2|-|81.60%|113.71|27.36|47.08|155.45|0.016
|SegFormer_B3|-|82.47%|142.97|47.24|62.70|154.68|0.017
|SegFormer_B4|-|82.38%|171.05|64.01|73.26|151.11|0.017
|SegFormer_B5|-|82.58%|199.68|84.61|84.34|147.92|0.016
|SETR-Naive|Vision Transformer|77.29%|620.94|303.37|201.26|145.76|0.016
|SETR-PUP|Vision Transformer|78.08%|727.46|307.24|212.22|147.05|0.016
|SETR-MLA|Vision Transformer|76.52%|633.88|307.05|204.87|145.87|0.015


<!-- |GINet|ResNet50_OS8|78.66%|463.36|55.87|-|-|-
|GINet|ResNet101_OS8|78.4%|618.95|74.91|-|-|-
|GSCNN|ResNet50_OS8|80.67%|385.50|39.47|-|-|- -->

## How to add new models to Benchmark
### Performance statistics
Set up a test environment according to the above configuration, and test according to the requirements of the test method. Among them, Inference Time(ms), Preprocess Time(ms), Postprocess Time(ms) can be tested through [PaddleSeg Inference Deployment Tutorial](deployment/inference/python_inference.md), and enable the `--benchmark` parameter for inference.

### Chart drawing
Update the obtained performance data to the table. The performance comparison chart drawing code is located in `PaddleSeg/tools/plot_model_performance.py`, supplement the performance data of the model in set_model_info(), run
````python
python plot_model_performance.py
````
All performance comparison charts are available
98 changes: 11 additions & 87 deletions docs/model_zoo_overview_cn.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,16 @@

# 模型库

## 模型库
### CNN系列
## 语义分割模型库

目前，PaddleSeg提供了40+语义分割模型、150+预训练模型、10+骨干网络。

在`PaddleSeg/configs`目录下，我们给出了所有模型在公开数据集上的配置文件和README文件，比如[PP-LiteSeg模型](../configs/pp_liteseg/)。
其中，README文件介绍了模型论文和训练精度，而且给出了公开数据集上训练的权重，可以直接下载进行测试。

如下我们列出了部分模型。

**CNN系列**

|模型\骨干网络|ResNet50|ResNet101|HRNetw18|HRNetw48|
|-|-|-|-|-|
Expand Down Expand Up @@ -49,92 +57,8 @@
|[GloRe](../configs/glore)|✔|-|-|-|
|[PP-LiteSeg](../configs/pp_liteseg)|-|-|-|-|

### Transformer系列
**Transformer系列**
* [SETR](../configs/setr)
* [MLATransformer](../contrib/AutoNUE/configs)
* [SegFormer](../configs/segformer)
* [SegMenter](../configs/segmenter)

# 模型库Benchmark
基于Cityscapes数据集，PaddleSeg支持22+系列分割算法以及对应的30+个图像分割预训练模型，性能评估如下。

**测试环境：**

- GPU: Tesla V100 16GB
- CPU: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
- CUDA: 10.2
- cuDNN: 7.6
- Paddle: 2.1.3
- PaddleSeg: 2.3

**测试方法:**

- 单GPU，Batch size为1，运行耗时为纯模型预测时间，预测图片尺寸为1024x512。
- 模型导出后使用Paddle Inference的Python API测试。
- 推理时间是使用CityScapes数据集中的100张图像进行预测取平均值的结果。
- 部分算法只测试了取得最高分割精度的配置下的模型性能。

## 精度 vs 速度
<div align="center">
<img src=https://user-images.githubusercontent.com/30695251/140323144-c44671ac-8ff8-4c11-a6f7-1ea339c27852.png //>
</div>

## 精度 vs FLOPs
<div align="center">
<img src=https://user-images.githubusercontent.com/30695251/140323107-02ce9de4-c8f4-4f18-88b2-59bd0055a70b.png //>
</div>

## 精度 vs Params
<div align="center">
<img src=https://user-images.githubusercontent.com/30695251/140323131-ed03fbb1-a583-47f5-a7dd-f4ea3582c345.png //>
</div>

## 总表
|Model|Backbone|mIoU|Flops(G)|Params(M)|Inference Time(ms)|Preprocess Time(ms)|Postprocess Time(ms)
|-|-|-|-|-|-|-|-|
|BiSeNetv2|-|73.19%|16.14|2.33|16.00|167.45|0.013
|Fast-SCNN|-|69.31%|2.04|1.44|10.43|161.52|0.012
|HarDNet|-|79.03%|35.40|4.13|21.19|164.36|0.013
|U-Net|-|65.00%|253.75|13.41|29.11|137.75|0.012
|SegFormer_B0|-|76.73%|13.63|3.72|15.66|152.60|0.017
|SegFormer_B1|-|78.35%|26.55|13.68|21.48|152.40|0.017
|STDC1-Seg50|STDC1|74.74%|24.83|8.29|9.10|153.01|0.016
|STDC2-Seg50|STDC2|77.60%|38.05|12.33|10.88|152.64|0.015
|ANN|ResNet101|79.50%|564.43|67.70|94.91|143.35|0.013
|DANet|ResNet50|80.27%|398.48|47.52|95.08|134.78|0.015
|Deeplabv3|ResNet101_OS8|80.85%|481.00|58.17|114|141.65|0.014
|Deeplabv3P|ResNet50_OS8|81.10%|228.44|26.79|69.78|147.24|0.016
|FCN|HRNet_W48|80.70%|187.50|65.94|45.46|130.58|0.012
|GCNet|ResNet101_OS8|81.01%|570.74|68.73|90.28|119.38|0.013
|OCRNet|HRNet_W48|82.15%|324.66|70.47|61.88|138.48|0.014
|PSPNet|ResNet101_OS8|80.48%|686.89|86.97|115.93|115.94|0.012
|DecoupledSegNet|ResNet50_OS8|81.26%|395.10|41.71|66.89|136.28|0.013
|EMANet|ResNet101_OS8|80.00%|512.18|61.45|80.05|140.47|0.013
|ISANet|ResNet101_OS8|80.10%|474.13|56.81|91.72|129.12|0.012
|DNLNet|ResNet101_OS8|81.03%|575.04|69.13|97.81|138.95|0.014
|SFNet|ResNet18_OS8|78.72%|136.80|13.81|69.51|131.67|0.015
|SFNet|ResNet50_OS8|81.49%|394.37|42.03|121.35|160.45|0.013
|PointRend|ResNet50_OS8|76.54%|363.17|28.18|70.35|157.24|0.016
|SegFormer_B2|-|81.60%|113.71|27.36|47.08|155.45|0.016
|SegFormer_B3|-|82.47%|142.97|47.24|62.70|154.68|0.017
|SegFormer_B4|-|82.38%|171.05|64.01|73.26|151.11|0.017
|SegFormer_B5|-|82.58%|199.68|84.61|84.34|147.92|0.016
|SETR-Naive|Vision Transformer|77.29%|620.94|303.37|201.26|145.76|0.016
|SETR-PUP|Vision Transformer|78.08%|727.46|307.24|212.22|147.05|0.016
|SETR-MLA|Vision Transformer|76.52%|633.88|307.05|204.87|145.87|0.015


<!-- |GINet|ResNet50_OS8|78.66%|463.36|55.87|-|-|-
|GINet|ResNet101_OS8|78.4%|618.95|74.91|-|-|-
|GSCNN|ResNet50_OS8|80.67%|385.50|39.47|-|-|- -->

## 如何添加新模型到Benchmark
### 性能数据统计
按照上述配置搭建测试环境，并按测试方法要求进行测试。其中Inference Time(ms)，Preprocess Time(ms)，Postprocess Time(ms)耗时可通过 [PaddleSeg推理部署教程](deployment/inference/python_inference_cn.md)进行测试，开启`--benchmark`参数进行推理即可。

### 图表绘制
将所获得的性能数据更新至表格中。性能对比图绘制代码位于`PaddleSeg/tools/plot_model_performance.py`，在set_model_info()中补充模型的性能数据，运行
```python
python plot_model_performance.py
```
即可得到所有的性能对比图