ParallelDo with MKLDNN

[pzelazko-intel]

1. In MKLDNN convolution (PR MKLDNN conv2d kernel added #8451) I need to send some data from forward to backward Compute functions. I'm saving it into DeviceContext. It works well as long as I'm not running it in ParallelDo mode. In ParallelDo, Compute method is called in different threads simuleanously with same DeviceContext, which is failing obviously because of memory access failures. How could I transport data from forward do backward safely?

2. I think that running MKLDNN Compute methods in parallel doesn't make sense, because MKLDNN uses internally OpenMP for parallel computations.
   For CPU place Paddle takes as many cores as possible: https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/get_places_op.cc#L51.
   We could change it so that when using MKLDNN, it would use only one device. But on the other hand, it won't be effective when we'll have both MKLDNN and GPU/Plain-CPU operators.
   It would be best if we could define device_count for each OP separately, but I don't know if it makes sense in the context of whole platform. Any thoughts?

[luotao1]

@tonyyang-svail @QiJune @jacquesqiao Can you help to answer it?

[QiJune]

@pzelazko-intel Yes, you are right. MKL will use OpenMP internally. And it has benefited from multi-cores already.

Parallel-Do module is not stable right now, and we are trying to profile and improve the performance of Parallel-Do. So the design and implement may be changed.

I advice we consider Parallel-Do MKL support after parallel-do module is stable.

[pzelazko-intel]

I was asking, because when I was using MKLDNN convolution, test_recognize_digits.py was failing in parallel test cases - https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/fluid/tests/book/test_recognize_digits.py#L80. MKLDNN is now disabled by default, so there is no problem at the moment.

We can close this issue and reopen it when Parallel-Do is stable.

[luotao1]

I close this issue due to the design of Paddle one-thread call MKLDNN multi-thread. And ParallelDo is replaced by ParallelExecutor now.