enable cpu adam op on powerpc architectures

[adammoody]

I am trying to build DeepSpeed on a PowerPC architecture (Power9). I run into two issues.

First, the -march option is not supported in gcc on this platform, though it seems like -mcpu could be a substitute for it. To support that, one change replaces that C++ flag.

Second, the cpuid.h and x86intrin.h headers do not exist on this system. To work around that, I've protected including those behind an x86 compile guard.

This allows the CPU_ADAM op to build, however, I don't have a good way to verify that the resulting build is actually valid.

[ghost]

All CLA requirements met.

[adammoody]

The asynchronous I/O op similarly has an -march=native flag in op_builder/async_io.py and includes for cpuid.h and x86intin.h in csrc/aio/py_lib/deepspeed_py_copy.h.

[tjruwase]

@adammoody, thanks so much for porting DeepSpeed to PowerPC, this is very important and greatly appreciated. In terms of validating the builds for both cpu_adam and aio, is it possible to run the unit tests? To get started you may want to initially focus on the specific unit tests for cpu_adam and aio.

[tjruwase]

@adammoody, this PR is failing CI now because of code formatting issues. Please see this. Thanks!

[adammoody]

@adammoody, thanks so much for porting DeepSpeed to PowerPC, this is very important and greatly appreciated.

Thanks. I'm happy to be help where I can.

In terms of validating the builds for both cpu_adam and aio, is it possible to run the unit tests? To get started you may want to initially focus on the specific unit tests for cpu_adam and aio.

Yes, I'll try to give this a shot.

[stas00]

Has this by chance changed something in the nvme extension build process, as our nvme test started failing to jit build after 0.4.3 release: huggingface/transformers#12715

[stas00]

update: false alarm, the problem was solved by rm -rf ~/.cache/torch_extensions/ as suggested by
@tjruwase

[adammoody]

@tjruwase , is this the proper way to run the cpu_adam test?

>>: pytest tests/unit/test_cpu_adam.py
============================================================ test session starts =============================================================
platform linux -- Python 3.7.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /path/to/DeepSpeed
collected 6 items                                                                                                                            

tests/unit/test_cpu_adam.py ......                                                                                                     [100%]

============================================================= 6 passed in 11.89s =============================================================
Config: alpha=0.001000, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1

[tjruwase]

Yes, it is. And it looks like all 6 tests passed.

Also, you can run verbose mode by adding -sv flags: pytest -sv tests/unit/test_cpu_adam.py.

[adammoody]

For the aio test, I get:

============================================================ test session starts =============================================================
platform linux -- Python 3.7.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /path/to/DeepSpeed
collected 28 items                                                                                                                           

tests/unit/test_aio.py ............................                                                                                    [100%]

======================================================= 28 passed in 129.68s (0:02:09) =======================================================

I'm guessing both of these might be slow compared to x86, since on PowerPC, both fall back to use a non-SIMD implementation. Can you judge performance from this?

[adammoody]

To get aio to actually build on my Redhat system, I also had to make this change:

diff --git a/op_builder/async_io.py b/op_builder/async_io.py
index 78aa2fe..9c6dde6 100644
--- a/op_builder/async_io.py
+++ b/op_builder/async_io.py
@@ -50,7 +50,7 @@ class AsyncIOBuilder(OpBuilder):
         return ['-laio']
 
     def is_compatible(self):
-        aio_libraries = ['libaio-dev']
+        aio_libraries = ['libaio-dev', 'libaio-devel']
         aio_compatible = self.libraries_installed(aio_libraries)
         if not aio_compatible:
             self.warning(
diff --git a/op_builder/builder.py b/op_builder/builder.py
index 3eeb4e4..bd6bfa8 100644
--- a/op_builder/builder.py
+++ b/op_builder/builder.py
@@ -151,9 +151,8 @@ class OpBuilder(ABC):
 
     def libraries_installed(self, libraries):
         valid = False
-        check_cmd = 'dpkg -l'
         for lib in libraries:
-            result = subprocess.Popen(f'dpkg -l {lib}',
+            result = subprocess.Popen(f'rpm -q -l {lib}',
                                       stdout=subprocess.PIPE,
                                       stderr=subprocess.PIPE,
                                       shell=True)

That's related to the issue @stas00 opened here: #1126

And actually, I have DeepSpeed compiling and linking to a libaio that is installed via conda, rather than the system install from Redhat. I just hacked the builder.py to find the Redhat rpm to enable the build.

[stas00]

I wrote the code to handle 3 flavors of linux here: #1126 (comment)
I think solution #2 would be the easiest in the long run IMHO.

@adammoody, if you'd like to integrate and PR it that would be super helpful! Thank you! That is if @tjruwase is in favor of using that solution and not others (or some other solutions).

[tjruwase]

Solution #2 sounds good to me, and such a PR would be greatly appreciated. Thanks @stas00 and @adammoody.

[tjruwase]

I'm guessing both of these might be slow compared to x86, since on PowerPC, both fall back to use a non-SIMD implementation. Can you judge performance from this?

SIMD perf is not important to aio as it is an NVMe library.

For cpu_adam, understanding perf implication is a bit more tricky as it heavily uses SIMD and MKL. So it might require experiments with multi-billion parameter models with offloading to get the practical impact. You might find these scripts useful for starters.

[adammoody]

Sure, I'll work on solution #2 to get started.

However, I suspect my conda setup doesn't quite fit properly with those three package managers, since my build isn't actually using the libaio from the rpm. Solution #4 to test a compile/link might work. Maybe something with compiler.has_function like described on this page would work?

https://www.cac.cornell.edu/wiki/index.php?title=Python_Distutils_Tips#How_to_find_if_a_library_exists

compiler=distutils.ccompiler.new_compiler()
if compiler.has_function('timer_create',libraries=('rt',)):
  user_macros.append(('HAVE_POSIX_TIMER','1'))

I can investigate that, as well. In my build of DeepSpeed, I'm externally setting CC, CXX, and CFLAGS to point to conda-specific items (like the include path to libaio.h) and that gets picked up in the build.

[stas00]

I think there should be an easy way to do this for a typical user, which is what approach 2 was trying to do and then allow anybody to do it in their own way by overriding the default checking.

The problem with solution 4 is that we can't tell the user what to do if it fails, so we still need to know what system they are on and what library to tell them to install - as you can see they are differently named on different flavors of linux.

Chances are very low that users will already have this library installed.

So it might be a combination of different solutions.

Also may I ask why the rpm library doesn't work for you? No sudo access to install it?

[adammoody]

Thanks @stas00 , I'll stick with the plan to implement solution 2.

In my case, I don't have sudo access on this particular system. The system admins have installed the libaio-devel rpm on the host system.

Having said that, in this case, I'm trying to avoid using the existing system install. That's because I'm using a base install of IBM's OpenCE for PyTorch and other software, which uses conda. Under this conda environment, it's best practice to only build against software installed within the same conda environment rather than using "external" system-installed packages. I don't think this will be a common problem for others, though other OpenCE users might be interested.

[stas00]

Thank you for explaining your particular needs.

This is great, so as suggested earlier the solution should include the generic automatic package-manager-based approach which should cater to the majority, and the manual solution for when either the automatic version doesn't cover the yet to be supported platform and when one doesn't want to use the former and wants to provide their own (your case). So there needs to be a way to tell the builder - I'm taking over, here is all the details you need.

[FarzanT]

Hi,
I wanted to add that a minor issue still persists on PowerPC architectures, cause by the following:

DeepSpeed/deepspeed/ops/adam/cpu_adam.py

Line 78 in 98cc35b

self.cpu_vendor = get_cpu_info()["vendor_id_raw"].lower()

Which results in KeyError: 'vendor_id_raw'.
I fixed it by simply replacing it with self.cpu_vensor="PowerPC". I guess the string doesn't really matter.
Should I create a new pull request?

[stas00]

Should I create a new pull request?

This is an excellent idea, @FarzanT