Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
问题
v100-16G环境下执行以下脚本,报OOM错误:
修复
修改PaddleSeg repo中的slim/prune.py文件,将
修改为:
https://github.com/PaddlePaddle/PaddleSeg/blob/develop/slim/prune.py#L152
已验证该修复之后,模型训练显存占用不会超过16G.
原因:
在构造L1NormFilterPruner实例时,会根据sample_shape执行一遍动态图的前向计算,以便将动态图转成静态图,该过程占用的显存与sample_shape正相关。动态图转成静态图是为了分析模型结构,所用的sample_shape不会影响分析的准确性,所以要尽量使用小的sample_shape.
以CityScape数据为例,
train_dataset
的shape为 (3, 512, 1024),val_dataset
的shape为(3, 1024, 2048) ;