"multiple definition of" linking error when building scikit-image with intel toolchain

[boegel]

Installing scikit-image-0.18.1-intel-2020b.eb has started failing for me recently, with:

  ld: /software/LibTIFF/4.1.0-GCCcore-10.2.0/lib/libtiff.so:(*IND*+0x0): multiple definition of `_end'
  ld: /software/LibTIFF/4.1.0-GCCcore-10.2.0/lib/libtiff.so:(*IND*+0x0): multiple definition of `__bss_start'
  error: command 'icpc' failed with exit status 1
  Building wheel for imread (setup.py): finished with status 'error'

It seems like the trigger for this problem are the changes made in #14079; the problem does not occur if LibTIFF was installed with the LibTIFF-4.1.0-GCCcore-10.2.0.eb that is included in EasyBuild v4.4.2 (which does not have the additional dependencies that were added in #14079).

This looks very similar to the problems reported in #6953, which never got resolved...
However, the --enable-ld-version-script was already there in LibTIFF-4.1.0-GCCcore-10.2.0.eb, it was not introduced with #14079.

[boegel]

https://bugs.gentoo.org/623566 suggests this has something to do with not using ld.gold, which could be triggered by --enable-ld-version-script (and maybe not always cause trouble)?

[boegel]

Making sure that libtiff.so is linked with ld.gold, by prepending export LD='ld.gold' && to the configure command, doesn't resolve the problem, although the LibTIFF configure script does pick up on $LD:

checking for ld used by gcc... ld.gold
checking if the linker (ld.gold) is GNU ld... yes

[boegel]

@smoors Any ideas on this?

[branfosj]

I'm surprised that we're not seeing more issues with imread. Looking at setup.py - starting at line 55 and going through to the definition of ext_modules on line 93. This hard codes include dirs to be /usr/local/include and /usr/X11/include and library dirs to be /usr/local/lib and /usr/X11/lib.

[zdemat]

Hi,

I can confirm the issue on {vis}[intelcuda/2020b] scikit-image v0.18.1 w/ Python 3.8.6 #12849 is the same as here, i.e. it is related to updated LibTIFF-4.1.0-GCCcore-10.2.0.eb.

[zdemat]

@branfosj and @boegel

I see I think more general issue, likely not only for imread. If I try a test code

\\test.cpp
#include "tiffio.h"

void test()
{
    TIFF* tif = TIFFOpen("foo.tif", "r");
    //... do stuff ...
    TIFFClose(tif);
}

module add GCCcore/10.2.0 libjpeg-turbo LibTIFF iccifort/2020.4.304
icpc -fpic -o test.o -c test.cpp

And the next step does not work like this

icpc -shared test.o -ljpeg -ltiff -o test.so
ld: /sw/tmp/z/s/software/LibTIFF/4.1.0-GCCcore-10.2.0/lib/libtiff.so:(*IND*+0x0): multiple definition of `_end'
ld: /sw/tmp/z/s/software/LibTIFF/4.1.0-GCCcore-10.2.0/lib/libtiff.so:(*IND*+0x0): multiple definition of `_edata'
ld: /sw/tmp/z/s/software/LibTIFF/4.1.0-GCCcore-10.2.0/lib/libtiff.so:(*IND*+0x0): multiple definition of `__bss_start'

but it works this way

icpc -shared test.o -ltiff -ljpeg -o test.so

when is swapped the order of -ljpeg and -ltiff

[zdemat]

It is in clear as LibTIFF depends on libjpeg-turbo. But it was like that also before the update of LibTIFF easyconfig.