When the horovod can support MXNet for training by allreduce?

[NanYan1119]

I heard that horovod will support MXNet for training by allreduce. Do you know when the version can use? I got the message from here: apache/mxnet#10696. @ctcyang

[ctcyang]

Hi,
We have a working prototype for the moment (https://github.com/ctcyang/horovod/tree/mxnet_fp16_divide_before_sum/examples/mxnet). However, we are in the process of redesigning the interface. If you don't want to change your code, it might be worthwhile to wait for the new API.

Our schedule is here: https://cwiki.apache.org/confluence/display/MXNET/Horovod-MXNet+Integration#Horovod-MXNetIntegration-Milestones

[NanYan1119]

@ctcyang Thank you! I will continue to pay attention to the schedule. By the way, I have a question of the horovod-mxnet.
Have you paid attention to the performance when the MXNet optimizer is LBSGD and SGD? Can the computation like lars when updating parameters in LBSGD and the backward computation overlap? Because when there is no server, all workers do the allreduce operation in GPU. It seems updating parameters and the backward computation can not overlap.

[ctcyang]

The parameter update and backward computation do overlap. @rahul003 says LBSGD seems slow, because some operations are done in numpy (i.e. not on GPU).

[NanYan1119]

@ctcyang @rahul003
Thank you! I also found that when using LBSGD, it needs to change the weights and grads (on GPU) which type is NDArray to the type in numpy(on CPU) using NDArray.asnumpy api, then do the norm operation use the api in numpy(on CPU). The NDArray.asnumpy api needs to copy from GPU to CPU by calling MXNDArraySyncCopyToCPU. This may lead to the LBSGD slow. So I think if using allreduce in horovod-mxnet(all workers on GPU and no server), the operations like LBSGD should on GPU not on CPU. Are you concerned about this?

[ctcyang]

Yes, we are definitely concerned about this. If you have the know-how and are interested in submitting a PR for GPU-only solution, we would definitely be interested!

[NanYan1119]

@ctcyang Sorry! I have tried to solve this problem, but I found it difficult to do.

[miaekim]

@ctcyang Hello, the doc says "Oct. 5, 2018: Beta release of Final API", if this is already released as beta, which version of Mxnet is supports this feature?

[ctcyang]

@miaekim It is merged into MXNet master branch already, so it will be part of MXNet 1.4.0 release, which will be announced in a few days and have pre-built packages. If you can't wait that long, you can build from source using master branch.

[miaekim]

@ctcyang Thank you so much