Fluid performance tuning plan

[reyoung]

We plan to tune fluid's performance with a loop with three steps:

1. Profile: To figure out which part of the fluid is slow.
2. Find problems & Give a fix: We will discuss and find the problems based on profile results.
3. Profile: To confirm the problems has been solved and the performance is improved.

There are several jobs for these three steps:

1. Find a machine with docker and GPU for profiling. @jacquesqiao
2. Neural network configurations for CNN, LSTM, etc. @qingqing01 @dzhwinter
   • Add benchmark configuration and scripts for convnets. #6056
3. Setup an environment for profiling. @chengduoZH
   • Use cProfile for Python, yap for Python/C++, nvprof for CUDA
4. Find problems: All members together.
5. Fix GPU problems: @jacquesqiao @qingqing01
6. Fix CPU problems: @dzhwinter
7. Fix Python problems: TODO

[dzhwinter]

1. Python code optimization
   We use CProfile profiling the python code. The tool usage can be checked out at cpu_profiling. If we need to dig into the code. You can add this snippet to the script, then you will get an accurate function call count. This is useful to measure whether there is redundant Variable, Operator is created.

import cProfile
pr = cProfile.Profile()
pr.enable()
// here the test codes. 
pr.disable()

• fine-tune the fit_a_line tests, check the components time cost. e.g.Program, Variable, Operator, Block
• fine-tune one forward pass, check total time cost in Python side.

2. C++ code optimization
   The yep + gperftools is excellent, but there is many Py_Eval, Py_Object function call disturb the focus point. Here is the method only retain necessary c++ calls.

#ifdef ENABLE_PROFILE
#include <gperftools/profiler.h>
#endif

ProfilerStart("cs.prof");
// here the test codes.
ProfilerStop()
ProfilerFlush()

• fine-tune the training forward/backward process, find the bottle-neck.

3. Cuda code
   TBD

[dzhwinter]

Also, we had created benchmark docker image and benchmark repo for every one reproduce the result.
The benchmark repo contains the tensorflow and paddle scripts.
https://github.com/dzhwinter/benchmark

The docker image contains all the profile tools.
https://hub.docker.com/r/dzhwinter/benchmark

Need to note that the docker image should be run in host mode, then you can check the profiler result in your browser. @tonyyang-svail

nvidia-docker run -it --rm --network=host -v $PWD:/paddle -v /root/.cache:/root/.cache  dzhwinter/benchmark /bin/bash