Avoid unused parameters assert by default

[tjruwase]

Unused parameters assert should be disabled by default

[tjruwase]

Changes behavior of #945.

[ghosthamlet]

@tjruwase Thanks for notification. I noticed you changed two things, I not very sure why that is better, can you explain?

1. disabled unused parameters assert. Why not just follow torch.nn.parallel.DistributedDataParallel's practice to enable it for easy debug? If you disabled it default, then most time it is hard for users to notice the unsed parameters in their models, then this option may be not very useful for debugging.
2. find_unused_parameters changed to ignore_unused_parameters. Why not follow torch.nn.parallel.DistributedDataParallel's naming practice?

[ghosthamlet]

Besides, I found you forgot to change the doc: https://github.com/microsoft/DeepSpeed/blob/master/docs/_pages/config-json.md#zero-optimizations-for-fp16-training.

[tjruwase]

1. disabled unused parameters assert. Why not just follow torch.nn.parallel.DistributedDataParallel's practice to enable it for easy debug? If you disabled it default, then most time it is hard for users to notice the unsed parameters in their models, then this option may be not very useful for debugging.

Enabling this by default will break dynamic networks such as g-shard and MoE, which train a subset of parameters in different iterations.

2. find_unused_parameters changed to ignore_unused_parameters. Why not follow torch.nn.parallel.DistributedDataParallel's naming practice?

I think these two are doing slightly different things. find_unused_parameters is False by default, but when enabled will traverse the graph to find unused parameters, incurring runtime overhead. ignore_unused_parameters is True by default to avoid false positive errors for dynamic networks, but also give static networks are way to detect unused parameters. The name difference reflects the fact that it does not involve graph traversing or finding with overheads.

[ghosthamlet]

@tjruwase Thanks for the detailed explanation, especially the difference of find_unused_parameters in Pytorch, I did not know that before.