error compiling openblas with older assembler

[pjmills]

hi All,
I am attempting to compile Julia as per instructions but receiving the following message at the end of the compilation

make[4]: *** [dgemm_kernel_HASWELL.o] Error 1
make[4]: *** Waiting for unfinished jobs....
make[3]: *** [libs] Error 1
*** Clean the OpenBLAS build with 'make -C deps clean-openblas'. Rebuild with 'make OPENBLAS_USE_THREAD=0 if OpenBLAS had trouble linking libpthread.so, and with 'make OPENBLAS_TARGET_ARCH=NEHALEM' if there were errors building SandyBridge support. Both these options can also be used simultaneously. ***
make[2]: *** [openblas-v0.2.10.rc2/libopenblas.so] Error 1
make[1]: *** [julia-release] Error 2
make: *** [release] Error 2

Could anyone help shed some light as to what the problem is please.

For Info - the OS is centos 6.5

Thanks

Peter

[jiahao]

Does the problem persist after cleaning the OpenBLAS build and remaking?

[pjmills]

Hi

yes it does.

Thanks

From: Jiahao Chen [mailto:notifications@github.com]
Sent: 18 July 2014 14:09
To: JuliaLang/julia
Cc: Peter Mills
Subject: Re: [julia] Compiling Issue (#7653)

Does the problem persist after cleaning the OpenBLAS build and remaking?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7653#issuecomment-49428414.

---

[http://www.plymouth.ac.uk/images/email_footer.gif]http://www.plymouth.ac.uk/worldclass

This email and any files with it are confidential and intended solely for the use of the recipient to whom it is addressed. If you are not the intended recipient then copying, distribution or other use of the information contained is strictly prohibited and you should not rely on it. If you have received this email in error please let the sender know immediately and delete it from your system(s). Internet emails are not necessarily secure. While we take every care, Plymouth University accepts no responsibility for viruses and it is your responsibility to scan emails and their attachments. Plymouth University does not accept responsibility for any changes made after it was sent. Nothing in this email or its attachments constitutes an order for goods or services unless accompanied by an official order form.

[jiahao]

What are the output lines just before make[4]: *** [dgemm_kernel_HASWELL.o] Error 1?

[pjmills]

Lines before:-

../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1709: Error: no such instruction: vpermpd $ 0xb1,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1709: Error: no such instruction:vpermpd $ 0x1b,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1709: Error: no such instruction: vpermpd $ 0xb1,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1711: Error: no such instruction:vpermpd $ 0xb1,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1711: Error: no such instruction: vpermpd $ 0x1b,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1711: Error: no such instruction:vpermpd $ 0xb1,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1712: Error: no such instruction: vpermpd $ 0xb1,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1712: Error: no such instruction:vpermpd $ 0x1b,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1712: Error: no such instruction: vpermpd $ 0xb1,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1713: Error: no such instruction:vpermpd $ 0xb1,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1713: Error: no such instruction: vpermpd $ 0x1b,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1713: Error: no such instruction:vpermpd $ 0xb1,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1714: Error: no such instruction: vpermpd $ 0xb1,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1714: Error: no such instruction:vpermpd $ 0x1b,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1714: Error: no such instruction: vpermpd $ 0xb1,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1734: Error: no such instruction:vpermpd $ 0xb1,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1734: Error: no such instruction: vpermpd $ 0x1b,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1734: Error: no such instruction:vpermpd $ 0xb1,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction: vpermpd $ 0xb1,%ymm5,%ymm5' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction:vpermpd $ 0xb1,%ymm7,%ymm7'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction: vpermpd $ 0x1b,%ymm2,%ymm2' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction:vpermpd $ 0x1b,%ymm3,%ymm3'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction: vpermpd $ 0xb1,%ymm2,%ymm2' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction:vpermpd $ 0xb1,%ymm3,%ymm3'

When I ran make -C deps clean-openblas

These are the lines at the end

../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1713: Error: no such instruction: vpermpd $ 0xb1,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1714: Error: no such instruction:vpermpd $ 0xb1,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1714: Error: no such instruction: vpermpd $ 0x1b,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1714: Error: no such instruction:vpermpd $ 0xb1,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1734: Error: no such instruction: vpermpd $ 0xb1,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1734: Error: no such instruction:vpermpd $ 0x1b,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1734: Error: no such instruction: vpermpd $ 0xb1,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction:vpermpd $ 0xb1,%ymm5,%ymm5'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction: vpermpd $ 0xb1,%ymm7,%ymm7' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction:vpermpd $ 0x1b,%ymm2,%ymm2'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction: vpermpd $ 0x1b,%ymm3,%ymm3' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction:vpermpd $ 0xb1,%ymm2,%ymm2'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction: `vpermpd $ 0xb1,%ymm3,%ymm3'
make[4]: *** [dgemm_kernel_HASWELL.o] Error 1
make[4]: *** Waiting for unfinished jobs....
make[3]: *** [libs] Error 1
*** Clean the OpenBLAS build with 'make -C deps clean-openblas'. Rebuild with 'make OPENBLAS_USE_THREAD=0 if OpenBLAS had trouble linking libpthread.so, and with 'make OPENBLAS_TARGET_ARCH=NEHALEM' if there were errors building SandyBridge support. Both these options can also be used simultaneously. ***
make[2]: *** [openblas-v0.2.10.rc2/libopenblas.so] Error 1
make[1]: *** [julia-release] Error 2
make: *** [release] Error 2

Which to me look the same.

Thanks for your help.

Peter

From: Jiahao Chen [mailto:notifications@github.com]
Sent: 18 July 2014 14:15
To: JuliaLang/julia
Cc: Peter Mills
Subject: Re: [julia] Compiling Issue (#7653)

What are the output lines just before make[4]: *** [dgemm_kernel_HASWELL.o] Error 1?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7653#issuecomment-49429001.

---

[http://www.plymouth.ac.uk/images/email_footer.gif]http://www.plymouth.ac.uk/worldclass

This email and any files with it are confidential and intended solely for the use of the recipient to whom it is addressed. If you are not the intended recipient then copying, distribution or other use of the information contained is strictly prohibited and you should not rely on it. If you have received this email in error please let the sender know immediately and delete it from your system(s). Internet emails are not necessarily secure. While we take every care, Plymouth University accepts no responsibility for viruses and it is your responsibility to scan emails and their attachments. Plymouth University does not accept responsibility for any changes made after it was sent. Nothing in this email or its attachments constitutes an order for goods or services unless accompanied by an official order form.

[simonster]

This usually means your assembler is too old to support Haswell instructions. See Linux build troubleshooting for 3 possible workarounds.

[tkelman]

@pjmills do you know what your processor model number is?

Others: Now that we've upgraded to OpenBLAS v0.2.10, we can also add a OPENBLAS_NO_AVX2 flag to be more specific here (or maybe a more generic OPENBLAS_FLAGS like we do for LLVM?). The Haswell kernels are the ones that require newer binutils (which can also be installed via the Scientific Linux devtoolset, I can't find the discussion off the top of my head but there's a link in another recent issue here), but OPENBLAS_NO_AVX will slow down Sandy Bridge too.

[pjmills]

Morning Tony,

Thanks for your reply.

I have checked the CPU info on my linux box and it show it as a single processor - Intel(R) Pentium(R) 4 CPU 2.80GHz.

I hope this helps.

Regards

Peter

processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Pentium(R) 4 CPU 2.80GHz
stepping : 1
cpu MHz : 2800.000
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc pebs bts pni dtes64 monitor ds_cpl cid cx16 xtpr
bogomips : 5600.13
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 48 bits virtual

From: Tony Kelman [mailto:notifications@github.com]
Sent: 18 July 2014 23:18
To: JuliaLang/julia
Cc: Peter Mills
Subject: Re: [julia] Compiling Issue (#7653)

@pjmillshttps://github.com/pjmills do you know what your processor model number is?

Others: Now that we've upgraded to OpenBLAS v0.2.10, we can also add a OPENBLAS_NO_AVX2 flag to be more specific here (or maybe a more generic OPENBLAS_FLAGS like we do for LLVM?). The Haswell kernels are the ones that require newer binutils (which can also be installed via the Scientific Linux devtoolset, I can't find the discussion off the top of my head but there's a link in another recent issue here), but OPENBLAS_NO_AVX will slow down Sandy Bridge too.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7653#issuecomment-49485868.

---

[http://www.plymouth.ac.uk/images/email_footer.gif]http://www.plymouth.ac.uk/worldclass

This email and any files with it are confidential and intended solely for the use of the recipient to whom it is addressed. If you are not the intended recipient then copying, distribution or other use of the information contained is strictly prohibited and you should not rely on it. If you have received this email in error please let the sender know immediately and delete it from your system(s). Internet emails are not necessarily secure. While we take every care, Plymouth University accepts no responsibility for viruses and it is your responsibility to scan emails and their attachments. Plymouth University does not accept responsibility for any changes made after it was sent. Nothing in this email or its attachments constitutes an order for goods or services unless accompanied by an official order form.

[tkelman]

Pentium 4, that's way too old to be worrying about AVX. Try

echo "OPENBLAS_NO_AVX = 1" >> Make.user
make

[pjmills]

Tony,

Many Thanks, I thought it might be a problem when I released it was an old machine.

But only using this machine as a test before I install on a cluster system

Will let you know how things go.

Regards

Peter

From: Tony Kelman [mailto:notifications@github.com]
Sent: 21 July 2014 08:19
To: JuliaLang/julia
Cc: Peter Mills
Subject: Re: [julia] error compiling openblas with older assembler (#7653)

Pentium 4, that's way too old to be worrying about AVX. Try

echo "OPENBLAS_NO_AVX = 1" >> Make.user

make

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7653#issuecomment-49577315.

---

[http://www.plymouth.ac.uk/images/email_footer.gif]http://www.plymouth.ac.uk/worldclass

This email and any files with it are confidential and intended solely for the use of the recipient to whom it is addressed. If you are not the intended recipient then copying, distribution or other use of the information contained is strictly prohibited and you should not rely on it. If you have received this email in error please let the sender know immediately and delete it from your system(s). Internet emails are not necessarily secure. While we take every care, Plymouth University accepts no responsibility for viruses and it is your responsibility to scan emails and their attachments. Plymouth University does not accept responsibility for any changes made after it was sent. Nothing in this email or its attachments constitutes an order for goods or services unless accompanied by an official order form.

[pjmills]

Hi Tony,

Have just finished compiling after using your suggestion and theis is the output:-

../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1713: Error: no such instruction: vpermpd $ 0xb1,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1713: Error: no such instruction:vpermpd $ 0x1b,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1713: Error: no such instruction: vpermpd $ 0xb1,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1714: Error: no such instruction:vpermpd $ 0xb1,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1714: Error: no such instruction: vpermpd $ 0x1b,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1714: Error: no such instruction:vpermpd $ 0xb1,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1734: Error: no such instruction: vpermpd $ 0xb1,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1734: Error: no such instruction:vpermpd $ 0x1b,%ymm0,%ymm0'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1734: Error: no such instruction: vpermpd $ 0xb1,%ymm0,%ymm0' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction:vpermpd $ 0xb1,%ymm5,%ymm5'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction: vpermpd $ 0xb1,%ymm7,%ymm7' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction:vpermpd $ 0x1b,%ymm2,%ymm2'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction: vpermpd $ 0x1b,%ymm3,%ymm3' ../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction:vpermpd $ 0xb1,%ymm2,%ymm2'
../kernel/x86_64/dgemm_kernel_4x4_haswell.S:1743: Error: no such instruction: `vpermpd $ 0xb1,%ymm3,%ymm3'
make[4]: *** [dgemm_kernel_HASWELL.o] Error 1
make[4]: *** Waiting for unfinished jobs....
make[3]: *** [libs] Error 1
*** Clean the OpenBLAS build with 'make -C deps clean-openblas'. Rebuild with 'make OPENBLAS_USE_THREAD=0 if OpenBLAS had trouble linking libpthread.so, and with 'make OPENBLAS_TARGET_ARCH=NEHALEM' if there were errors building SandyBridge support. Both these options can also be used simultaneously. ***
make[2]: *** [openblas-v0.2.10/libopenblas.so] Error 1
make[1]: *** [julia-release] Error 2
make: *** [release] Error 2

Still seems to have a problem – Any thoughts on why it would fail?
Is it that the hardware really is too old for the software?

May give it a go directly on the cluster as this is newer hardware.

Peter
From: Tony Kelman [mailto:notifications@github.com]
Sent: 21 July 2014 08:19
To: JuliaLang/julia
Cc: Peter Mills
Subject: Re: [julia] error compiling openblas with older assembler (#7653)

Pentium 4, that's way too old to be worrying about AVX. Try

echo "OPENBLAS_NO_AVX = 1" >> Make.user

make

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7653#issuecomment-49577315.

---

[http://www.plymouth.ac.uk/images/email_footer.gif]http://www.plymouth.ac.uk/worldclass

This email and any files with it are confidential and intended solely for the use of the recipient to whom it is addressed. If you are not the intended recipient then copying, distribution or other use of the information contained is strictly prohibited and you should not rely on it. If you have received this email in error please let the sender know immediately and delete it from your system(s). Internet emails are not necessarily secure. While we take every care, Plymouth University accepts no responsibility for viruses and it is your responsibility to scan emails and their attachments. Plymouth University does not accept responsibility for any changes made after it was sent. Nothing in this email or its attachments constitutes an order for goods or services unless accompanied by an official order form.

[tkelman]

Odd, you may need to clean out the older openblas build via make -C deps distclean-openblas first in order for the flags to properly take effect.

Technically the old software of CentOS is at fault more than the Pentium 4 hardware. As long as you're running on something newer than an i486 processor then Julia should work.

[pjmills]

Hi Tony,

I have just run a test install on our cluster and this seemed to compile fine.

Just one question I put in the Make.user file that I wanted it to install to the /opt folder using the prefix option.

However, because I downloaded into a tmp folder and did the make here the software compiled and then I then ran Make install.

This did create a new folder in /opt/Julia with a Julia executable in /opt/Julia/bin

But now I have two Julia exectuables 1) in tmp/Julia and a second one in /opt/Julia/bin

Should I have done the installation a different way?
As this just seems to be duplicating files.

Thanks

Peter

From: Tony Kelman [mailto:notifications@github.com]
Sent: 21 July 2014 08:19
To: JuliaLang/julia
Cc: Peter Mills
Subject: Re: [julia] error compiling openblas with older assembler (#7653)

Pentium 4, that's way too old to be worrying about AVX. Try

echo "OPENBLAS_NO_AVX = 1" >> Make.user

make

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7653#issuecomment-49577315.

---

[http://www.plymouth.ac.uk/images/email_footer.gif]http://www.plymouth.ac.uk/worldclass

This email and any files with it are confidential and intended solely for the use of the recipient to whom it is addressed. If you are not the intended recipient then copying, distribution or other use of the information contained is strictly prohibited and you should not rely on it. If you have received this email in error please let the sender know immediately and delete it from your system(s). Internet emails are not necessarily secure. While we take every care, Plymouth University accepts no responsibility for viruses and it is your responsibility to scan emails and their attachments. Plymouth University does not accept responsibility for any changes made after it was sent. Nothing in this email or its attachments constitutes an order for goods or services unless accompanied by an official order form.

[tkelman]

Julia always creates an executable in the location where it gets compiled. If you install to a separate location then you should be able to move around or remove the source build and the installed copy should keep working. However for the sake of faster incremental builds to keep up with future updates to Julia, I would leave the source build where it is. You can do git pull && make in the future without having to rebuild all the large dependencies.