[Discussion]How can we get the dynamic number of trainers when we implement auto-scaling?

[gongweibao]

Now we get fan_in from receive operator's attribute.It's okay when the trainer's number is static. https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/operators/recv_op.cc#L97

auto fan_in = Attr<int>("Fanin");

How can we get the dynamic number of trainers when we implement auto-scaling?

[helinwang]

I can imagine two ways to support dynamic scaling:

• support dynamic reload of the ProgramDesc: but we still keep the scope / Vars (scope is similar to the container concept in distributed TF). In this way we can reload a new ProgramDesc with the correct fan in count. Or,
• in the recv OP implementation it can adapt to variant number of trainers using a timeout window and trainer id. If gradient with a new trainer id arrived, increment fan_in. If gradient did not arrive in a window, decrease fan_in and continue. However this approach does not support autoscaling the number of pservers, because it needs the ProgramDesc for the psever to change.

[gongweibao]

support dynamic reload of the ProgramDesc: but we still keep the scope / Vars (scope is similar to the container concept in distributed TF). In this way we can reload a new ProgramDesc with the correct fan in count

This one is interesting.

[putcn]

I agree with @helinwang on the 1st option, since ultimately we want to support transpile ProgramDesc into arbitrary pieces.
to implement this, I think we can borrow the concept of reconciliation from Kubernetes. When the ProgramDesc is transpiled into new shape or any parameter changed in deployment, in this case, trainer count changed, the whole system starts to reconcile into a new desired state.