Conditional clearing of MKL-DNN cache in Executor's destructor

[grygielski]

In current state, MKL-DNN cache is being cleared on destruction of an Executor object. This mechanism was introduced by #20241 to reduce the length of hashing keys.

However, it appeared to be a problem, especially during training models with e.g. conditional_op or similar ops that create their own Executor object inside. Upon destruction of this local Executor, whole MKL-DNN cache is being cleared, thus resulting in crashes during training or slow-downs during inference.

The solution would be to somehow distinguish these 2 "types" of Executors where one is created to execute the whole program (therefore clear mkl-dnn cache on destruction) and the other is created locally in some operators (which doesn't clear mkl-dnn cache when destroyed).

[grygielski]

PRs that rely on this Issue:

• support mnist and resnet dygraph_to_static test #25774 - test_resnet failing because of cache clearing in conditional_block_op
• don't clear mkldnn cache in block_op executor dtor #25735 - tries to solve this problem but solution does not satisfy

[grygielski]

#26502 introduced mechanism for blocking oneDNN cache clearing. For now it's implemented just for run_program_op, but can be easily extended to other operators with local executor.

[lidanqing-intel]

Close since PR merged