interpreter fails with assertion in debug builds on ARM when upgrading gcc

[pikacic]

Check duplicate issues.

• Checked for duplicates

Description

Using ROOT 6.30/04 built by EP-SFT stacks for aarch64-el9-gcc13-dbg on lxplus-arm.cern.ch the application is aborted because of a failed assert:

root.exe: /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.30.04/src/ROOT/6.30.04/interpreter/llvm-project/clang/include/clang/Serialization/ModuleFile.h:77: clang::serialization::InputFile::InputFile(clang::FileEntryRef, bool, bool): Assertion `!(isOverridden && isOutOfDate) && "an overridden cannot be out-of-date"' failed.

It turns out that the failure is implicitly caused by an updated version of GCC (from 11.3 to 11.4).

Reproducer

Here is a redacted log a sessions exposing the problem:

ssh lxplus-arm.cern.ch
>  rpm -q gcc
gcc-11.4.1-2.1.el9.aarch64
> . /cvmfs/sft.cern.ch/lcg/releases/gcc/13.1.0-b3d18/aarch64-el9/setup.sh
> . /cvmfs/sft.cern.ch/lcg/views/LCG_105a/aarch64-el9-gcc13-dbg/setup.sh
> root
root.exe: /build/jenkins/workspace/lcg_release_pipeline/build/projects/ROOT-6.30.04/src/ROOT/6.30.04/interpreter/llvm-project/clang/include/clang/Serialization/ModuleFile.h:77: clang::serialization::InputFile::InputFile(clang::FileEntryRef, bool, bool): Assertion `!(isOverridden && isOutOfDate) && "an overridden cannot be out-of-date"' failed.
>

The optimized build, of course, works

ssh lxplus-arm.cern.ch
>  rpm -q gcc
gcc-11.4.1-2.1.el9.aarch64
> . /cvmfs/sft.cern.ch/lcg/releases/gcc/13.1.0-b3d18/aarch64-el9/setup.sh
> . /cvmfs/sft.cern.ch/lcg/views/LCG_105a/aarch64-el9-gcc13-opt/setup.sh
> root
   ------------------------------------------------------------------
  | Welcome to ROOT 6.30/04                        https://root.cern |
  | (c) 1995-2024, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxarm64 on Feb 03 2024, 21:41:03                    |
  | From heads/master@tags/v6-30-04                                  |
  | With g++ (GCC) 13.1.0                                            |
  | Try '.help'/'.?', '.demo', '.license', '.credits', '.quit'/'.q'  |
   ------------------------------------------------------------------

root [0] .q
>

It woks also if I use an old container with gcc 11.3:

ssh lxplus-arm.cern.ch
> /cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/3067/stable/linux-aarch64/bin/apptainer shell -e -B /cvmfs /cvmfs/lhcb.cern.ch/containers/os-base/alma9-devel/2023-06-30_12-00-00/aarch64
WARNING: /cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/3067/stable/linux-aarch64/etc/singularity/ exists, migration to apptainer by system administrator is not complete
Apptainer> rpm -q gcc
gcc-11.3.1-4.3.el9.alma.aarch64
Apptainer> . /cvmfs/sft.cern.ch/lcg/releases/gcc/13.1.0-b3d18/aarch64-el9/setup.sh
Apptainer> . /cvmfs/sft.cern.ch/lcg/views/LCG_105a/aarch64-el9-gcc13-dbg/setup.sh
Apptainer> root
   ------------------------------------------------------------------
  | Welcome to ROOT 6.30/04                        https://root.cern |
  | (c) 1995-2024, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxarm64 on Feb 04 2024, 02:07:16                    |
  | From heads/master@tags/v6-30-04                                  |
  | With g++ (GCC) 13.1.0                                            |
  | Try '.help'/'.?', '.demo', '.license', '.credits', '.quit'/'.q'  |
   ------------------------------------------------------------------

root [0] .q
Apptainer>

ROOT version

   ------------------------------------------------------------------
  | Welcome to ROOT 6.30/04                        https://root.cern |
  | (c) 1995-2024, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxarm64 on Feb 04 2024, 02:07:16                    |
  | From heads/master@tags/v6-30-04                                  |
  | With g++ (GCC) 13.1.0                                            |
  | Try '.help'/'.?', '.demo', '.license', '.credits', '.quit'/'.q'  |
   ------------------------------------------------------------------

Installation method

SFTR LCG view

Operating system

RHEL9, Almalinux 9

Additional context

No response

[vgvassilev]

Is this error still reproducing with a build from scratch?

[pikacic]

The problem is present in SFT nightly builds too (at least in a build from Friday 2024-02-24).

I know they build from Docker containers and I'll check with them which version of system gcc is installed in their images so that we can confirm that the problem comes from an inconsistency between build time and run time.

[andresailer]

The problem seems to be an update of glibc-devel, in particular this file /usr/include/bits/floatn.h (it is the last file that is loaded before the assertion according to strace)

lxplus-arm

glibc-devel-2.34-83.el9_3.7.aarch64 : Object files for development using standard C libraries.
Repo        : @System
Matched from:
Filename    : /usr/include/bits/floatn.h

Our docker container

glibc-devel-2.34-60.el9_2.7.aarch64 : Object files for development using standard C libraries.
Repo        : @System
Matched from:
Filename    : /usr/include/bits/floatn.h

Updating glibc-devel in our docker container reproduces the issue. Updating gcc does not reproduce the issue.

There are some differences in the file like

- #  if !__GNUC_PREREQ (7, 0) || defined __cplusplus
+ #  if !__GNUC_PREREQ (7, 0) || (defined __cplusplus && !__GNUC_PREREQ (13, 0))

[dpiparo]

Fixed in 6.32 and LCG_106 ( . /cvmfs/sft.cern.ch/lcg/views/LCG_106/aarch64-el9-gcc13-dbg/setup.sh)