Repro-Build: jdk-18 debuginfo symbol files contain full path to build path making non-reproducible

[andrew-m-leonard]

debuginfo symbols contain the build object path including the full build path, this is thus not unique.

For gcc needs use of -fdebug-prefix-map to map "$WORKSPACE" to "." so paths are deterministic.
https://github.com/andrew-m-leonard/jdk-1/commit/a7458c76de9c3c5340c26af57d23b69860749cd2

[andrew-m-leonard]

Ref #2594

[andrew-m-leonard]

Also note that system headers also get referenced from the symbols, thus paths like "/usr/include/.." are embedded, but this is obviously more deterministic as this path is consistent on most platforms.

[magicus]

Check out FILE_MACRO_CFLAGS in make/autoconf/flags-cflags.m4. I think this flag can be baked into the same variable (and maybe renamed to PATH_MAP_CFLAGS instead)

[andrew-m-leonard]

Check out FILE_MACRO_CFLAGS in make/autoconf/flags-cflags.m4. I think this flag can be baked into the same variable (and maybe renamed to PATH_MAP_CFLAGS instead)

Thanks @magicus , yes one reason i've not tried that is i'm building with gcc7.5, -fmacro-prefix-map is only available gcc8+.

[magicus]

But -fdebug-prefix-map is available in gcc 7? Anyway, you could append to PATH_MAP_CFLAGS each of them separately.

[andrew-m-leonard]

But -fdebug-prefix-map is available in gcc 7? Anyway, you could append to PATH_MAP_CFLAGS each of them separately.

yes! However, we're going to look at moving up to gcc8+
But the problem in having to use -fmacro-prefix-map is actually resolved by the use of relative paths when we do the native compile it seems.

I also did discover a debug symbol issue that is not resolved by either of these, when dumping the debug symbols file, it seems "certain" object files xxx.o are embedded in the debug file with the full build path that they were compile with, the only way I found of fixing it was using the relative object path section that currently is only used for clang and >500 objs:
https://github.com/andrew-m-leonard/jdk-1/blob/a7458c76de9c3c5340c26af57d23b69860749cd2/make/common/NativeCompilation.gmk#L1137

    # When linking a large amount of object files, we risk hitting the limit
    # of the command line length even on posix systems if the path length of
    # the output dir is very long due to our use of absolute paths. To
    # mitigate this, use paths relative to the output dir when linking over
    # 500 files with clang and the output dir path is deep.
    #ifneq ($$(word 500, $$($1_ALL_OBJS)), )
    #  ifeq ($$(TOOLCHAIN_TYPE), clang)
    #    # There is no strlen function in make, but checking path depth is a
    #    # reasonable approximation.
    #    ifneq ($$(word 10, $$(subst /, ,$$(OUTPUTDIR))), )
    $1_LINK_OBJS_RELATIVE := true
    $1_ALL_OBJS_RELATIVE := $$(patsubst $$(OUTPUTDIR)/%, %, $$($1_ALL_OBJS))
    #    endif
    #  endif

[andrew-m-leonard]

@magicus can you see any reason why we couldn't always use the $1_LINK_OBJS_RELATIVE when --with-reproducible_build is "true" ?

[magicus]

We have a policy to not use relative paths in the build system. All command lines should be "copy paste-able" from any point in the directory hierarchy. The relative linking above was a work-around for systems which cloned the repo in a deep path resulting in a command line limit.

We could of course use it for --enable-reproducible-builds, but that'd also be a bit counter towards the goal I have to be able to do all reproducible stuff by default, and retire --enable-reproducible-builds. We will never do relative paths by default...

[andrew-m-leonard]

@magicus That could be an interesting challenge, it's possible gcc 8+ might have the necessary options, i'll have to upgrade and give it a go. With gcc7.5 rather oddly there was always a single full path object file listed in the symbols data, I think it was src/hotspot/os_cpu/linux_x86/linux_x86_64.o which is mixed C + asm, not sure if that has any significance, maybe the mapping options aren't enabled for those...?

[magicus]

gcc 7 is starting to get old; my interest in supporting it beyond "keep it working, if possible" is minimal. I'd rather focus on getting this working The Right Way using modern gcc's.

[sxa]

FYI We have GCC 10.3 on the build systems now, and #2824 will enable it for Temurin JDK18+ for now (IMHO with less than a week to go before the January update cycle It's too close to consider doing it for earlier versions!)

[andrew-m-leonard]

I have not recreated this with gcc 10.3 on latest jdk upstream.