DLRM with ipex 1.10

[Peach-He]

Hi,
Not sure if you have plan on upgrading DLRM code to ipex 1.10. I tried to upgrade the DLRM code with ipex 1.10 based on the patch from https://github.com/intel/intel-extension-for-pytorch/blob/0.2/torch_patches/models/0001-enable-dlrm-distributed-training-for-cpu.patch and noticed performance regression.
Micro benchmark showed that all_to_all had 2x worse performance after upgrading ipex 1.10. Any idea?

system config:

• torch ccl 1.10, pytorch 1.10, ipex 1.10
• single node, 2 ranks per node

all2all ipex v0.2:

all2all ipex 1.10:

test code:

import torch
import os

import extend_distributed as ext_dist

if __name__ == "__main__":
    ext_dist.init_distributed(backend='ccl')
    inputs = []
    tensor1 = torch.ones(262144, 16, dtype=torch.bfloat16)
    tensor2 = torch.ones(262144, 16, dtype=torch.bfloat16)
    inputs.append(tensor1)
    inputs.append(tensor2)
    with torch.autograd.profiler.profile(True) as prof:
        for _ in range(10):
            a2a_req = ext_dist.alltoall(inputs, None)
            ly_sparse = a2a_req.wait()
    print(prof.key_averages().table(sort_by="cpu_time_total"))

Thanks

[EikanWang]

@Peach-He , yes, we have a plan to upgrade DLRM for IPEX 1.10. The work is WIP. @zhuhaozhe , please keep Peach posted if you have any updates.

[zhuhaozhe]

@Peach-He. We have updated DLRM for IPEX 1.10 for single-node training. And also enabled the functionality for distribute-training with 2 sockets on single-node. But the performance analysis is WIP. @liangan1 is working on distributed training on multi-node with performance tuning.

Hi, @liangan1. Do you have any insights on all2all performance?

[EikanWang]

@liangan1, could you please provide your feedback here?

[liangan1]

@EikanWang got it. let me do some experiments and will relay it later.

[liangan1]

sorry, @Peach-He @EikanWang I can't reproduce your performance result. The following is the performance data with 2s on clx-8280.
Linux kernel: Linux mlt2-clx111 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

**all2all ipex v0.2(with BKC ):**

all2all ipex 1.10:

[EikanWang]

Please feel free to reopen it if you still encounter this issue.