Make optimizer program as a block instead of the attribute of Recv op

[Yancey0623]

For the latest implement about Fluid distributed training, the optimizer program will be passed into Recv op as an attribute:

AddAttr<std::string>("OptimizeProgram", "type string",
                         "Serialized ProgramDesc string for recv to run.");

Maybe we could append the optimizer program into the block list and pass the block id into RecvOp, I think there is two advantage:

• All the variables will be infershaped before running.
• It's easy to debug, we can not print the whole pserver program after optimizer program be serialized to string.

[helinwang]

Agree! I faced the second issue today: #7419

[helinwang]

All the variables will be infershaped before running.

Curious about this part of the code, if you are familiar with the infershape code, can you point me the code that explains why "All the variables will be infershaped before running."? Thanks!

[helinwang]

@Yancey1989 @typhoonzero if you guys agree this is a good idea, and are not working on this, I can take this item.

I think we need to be able to print OptimizeProgram as plain text for debugging and unit test purposes.

[helinwang]

Just took a look, seems OP does not know about the current running ProgramDesc though. Maybe we should expose it? We are working around it by setting attribute anyway.
This lead another question: are we using any block other than the first block of the ProgramDesc anywhere at all?

CC: @reyoung @tonyyang-svail what is you guys' opinion? Probably while OP, recurrent OP have the same need.

[Yancey0623]

All the variables will be infershaped before running

I think exe.run(default_startup_program()) will do that.

[typhoonzero]

Totally agree, this change can also distributed transpiler do not create variables twice both in pserver program and recv op subprogram.

[typhoonzero]

@Yancey1989 @typhoonzero if you guys agree this is a good idea, and are not working on this, I can take this item.

@helinwang I'm not working on this currently.

[Yancey0623]

Thanks @typhoonzero @helinwang .

[helinwang]

Blocking issue: #7468