[BUG] flops-profiler latency is 1000x larger than actual value

[zionwu]

Describe the bug
We use the Deepspeed flops-profiler, but the result is wrong. The latency is 1000x larger than actual value and the gpu thoughput is 1000x smalller. After we checked the code and add some debug message, we found it is because it uses the wrong unit for lantency.

In https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/profiling/flops_profiler/profiler.py#L303 :

fwd_latency = self.ds_engine.timers('forward').elapsed(False)

the unit of the return value of elapsed() is microsecond, but the code treats it as second.

The image shows the debug message we added, the elapsed time is CudaEventTimer, which is in microsecond.

ds_report output

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [YES] ...... [OKAY]
cpu_adagrad ............ [YES] ...... [OKAY]
fused_adam ............. [YES] ...... [OKAY]
fused_lamb ............. [YES] ...... [OKAY]
sparse_attn ............ [YES] ...... [OKAY]
transformer ............ [YES] ...... [OKAY]
stochastic_transformer . [YES] ...... [OKAY]
async_io ............... [YES] ...... [OKAY]
utils .................. [YES] ...... [OKAY]
quantizer .............. [YES] ...... [OKAY]
transformer_inference .. [YES] ...... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/opt/conda/lib/python3.8/site-packages/torch']
torch version .................... 1.11.0+cu113
torch cuda version ............... 11.3
torch hip version ................ None
nvcc version ..................... 11.3
deepspeed install path ........... ['/opt/conda/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.6.5, unknown, unknown
deepspeed wheel compiled w. ...... torch 1.11, cuda 11.3

[tjruwase]

@zionwu, thanks for reporting this issue. I think you have correctly diagnosed the problem and so I am wondering if you are willing to contribute a PR. Thanks!

@cli99, FYI.

[zionwu]

@tjruwase yes, I will submit a PR for it. Thanks.