ROOT 6.36.04 does a factor 2 more stats when loading a library than 6.32.08

[ktf]

Check duplicate issues.

• Checked for duplicates

Description

We see a large number of (unneeded?) stats when trying to load a library. This results in greatly reduced performance on startup to the point it prevents us to start our datataking workflows.

In particular doing:

strace -f root -q -e 'gSystem->Load("libO2DataFormatFV0s.so")' 2>&1 | grep BUILD

results in 15843 matches while with the old release we have 6049. Notice in both cases none of the files actually exists. So this is really ROOT looking up in some unrelocated path, which itself is wrong.

Moreover it looks like the same stat is done multiple times. In both logs I have patterns like:

→ cat stats-new.txt | head -n 100
 newfstatat(AT_FDCWD, "/home/alibuild/sw/BUILD/<build-hash>/ROOT/lib/libc.pcm", 0x7ffd02cf75e0, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include", 0x7ffd02cf5fc0, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include", 0x7ffd02cf6430, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/stddef.h", 0x7ffd02cf61c0, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/stddef.h", 0x7ffd02cf6630, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/__stddef_size_t.h", 0x7ffd02cf6130, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/__stddef_size_t.h", 0x7ffd02cf65a0, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/__stddef_size_t.h", 0x7ffd02cf61c0, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/__stddef_size_t.h", 0x7ffd02cf6630, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/__stddef_null.h", 0x7ffd02cf6130, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/__stddef_null.h", 0x7ffd02cf65a0, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/__stddef_null.h", 0x7ffd02cf61c0, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/__stddef_null.h", 0x7ffd02cf6630, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/stdint.h", 0x7ffd02cf6130, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/stdint.h", 0x7ffd02cf65a0, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/stdint.h", 0x7ffd02cf61c0, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/stdint.h", 0x7ffd02cf6630, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/stdarg.h", 0x7ffd02cf6130, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/stdarg.h", 0x7ffd02cf65a0, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/stdarg.h", 0x7ffd02cf61c0, 0) = -1 ENOENT (No such file or directory)
 newfstatat(AT_FDCWD, "/data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/stdarg.h", 0x7ffd02cf6630, 0) = -1 ENOENT (No such file or directory)

which show repetitive, failing stats for the same file.

Reproducer

See above

ROOT version

6.36.04-alice2

Installation method

aliBuild

Operating system

Linux

Additional context

No response

[dpiparo]

Here another place where we see stats #14296 .

[dpiparo]

@ktf do you see the paths encoded in the library, i.e. can you show the path data/sw/BUILD/<build-hash>/ROOT/etc/cling/lib/clang/18/include/ in the library libO2DataFormatFV0s.so with the strings command?

[ktf]

Actually, I do:

[pdp@alio2-cr1-qc06 ~]$ strings /opt/o2/lib/libO2DataFormatsFV0.so | grep BUILD
/home/alibuild/sw/BUILD/7dfefc80e207f4fc7884ab4ef983296ca64c87e6/O2/DataFormats/Detectors/FIT/FV0
/home/alibuild/sw/BUILD/7dfefc80e207f4fc7884ab4ef983296ca64c87e6/O2/DataFormats/Detectors/FIT/FV0/src
/data/sw/BUILD/7dfefc80e207f4fc7884ab4ef983296ca64c87e6/O2/DataFormats/Detectors/FIT/FV0/
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init2039
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init1584
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init1909
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init86
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init1974
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init129
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init193
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init1519
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init225
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init257
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init289
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init161
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init1649
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init321
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init353
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init385
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init1714
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init1779
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init423
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init466
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init498
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init1844
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0ReadRules_G__O2DataFormatsFV02144
dIhomedIalibuilddIswdIBUILDdI7dfefc80e207f4fc7884ab4ef983296ca64c87e6dIO2dIDataFormatsdIDetectorsdIFITdIFV0dIG__O2DataFormatsFV0Init2104
/home/alibuild/sw/BUILD/7dfefc80e207f4fc7884ab4ef983296ca64c87e6/O2
/home/alibuild/sw/BUILD/7dfefc80e207f4fc7884ab4ef983296ca64c87e6/O2/DataFormats/Detectors/FIT/FV0/G__O2DataFormatsFV0.cxx
/home/alibuild/sw/BUILD/7dfefc80e207f4fc7884ab4ef983296ca64c87e6/O2/DataFormats/Detectors/FIT/FV0

[ferdymercury]

Also related: #12462

[ktf]

Not the one you asked for, though... I assume that comes from ROOT?

[ktf]

Indeed it looks like #14296 is the actual culprit, the new ROOT merely looks for way more things.

[ktf]

That said, why checking so many times?

[ktf]

I managed to make in ALICE-agnostic. It's enough to do:

strace -f root -q -e 'gSystem->Load("libCore.so")' 2>&1 | grep newfstatat | wc -l

to see that in one case you have 10K stats (which is already quite incredible) while the new vesion has over 20K.

[Barthelemy]

Hi,
I just want to stress the fact that it is a real issue for us (ALICE DAQ). It now takes so long to start processes that we time out. We can increase the timeout from 30 seconds to 45 seconds, but this is becoming very high. Is there a way at least to avoid the duplicate calls to stat ?