Chromium database much smaller than it should be

[Ludgershall]

Hello,

I've created a CodeQL database of Chromium. I've used the official tutorial from Google (https://chromium.googlesource.com/chromium/src/+/main/docs/linux/build_instructions.md).
When it came to building Chromium, I've swapped the command autoninja -C out/Default chrome by codeql database create chromium-database --language=cpp --command="autoninja -C out/Default chrome" -J-Xmx26034M.
EDIT: Just to be clear, I've started compiling from scratch.

Everything built just fine but when I took a look at the final database, it looked a lot smaller that I think it should be (a folder of 268 items for 762MB).
I've opened the resulting DB in VSCode and sure enough plenty of files were missing.

I've included a picture of the available files/folders present when opening the DB in VSCode.

Can you point out a way to have a more complete database? Thanks

Configuration:
Ubuntu 20
CodeQL CLI v2.8.5

[aibaars]

Thanks for the report @Ludgershall . To help debugging this issue, could you have a look in the log/build-tracer.log file for error messages that could explain what's going wrong? The file is probably too large to post as an attachment. It may also contain sensitive data (user name, paths) you might not want to share. Could you look for the phrases intercepted , /extractor and catastrophic error . This should give an idea of which processes were intercepted by CodeQL. For the analysis the work CodeQL needs to intercept all compiler calls (eg clang, gcc) and invoke the CodeQL C/C++ extractor from each of them to extract the information from the compiled source files and store them in the CodeQL database. The phrase catastrophic error usually indicates something going very wrong causing CodeQL to skip some source files. If you can post any error messages or other relevant information to this issue that would be greatly appreciated. Make sure to redact any sensitive information (in file paths) you don't want to share. Thanks!

[pooneh-nb]

I can share the build-tracer.log here. It's uploaded to my github.

[aibaars]

Thanks @pooneh-nb . I see there are over 35000 catastrophic error messages like language modes specified are incompatible. I'll pass the log file on to the team so they can have a closer look.

@Ludgershall could you have a look at the build-tracer.log on your system as well and check if you're seeing a similar failure pattern?

[Ludgershall]

I've got plenty of errors like the following

1 catastrophic error detected in this compilation.
Compilation terminated.
Current directory: /home/micah/chromium/src/out/Default
Command: /home/micah/Downloads/codeql/cpp/tools/linux64/extractor --mimic '/home/micah/chromium/src/third_party/llvm-build/Release+Asserts/bin/clang' -MMD -MF 'obj/buildtools/third_party/libc++/libc++/vector.o.d' -DLIBCXX_BUILDING_LIBCXXABI -DDCHECK_ALWAYS_ON=1 -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_OZONE=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_GNU_SOURCE '-DCR_CLANG_REVISION="llvmorg-15-init-7040-g50de659a-3"' -DCOMPONENT_BUILD -D_LIBCPP_ABI_UNSTABLE -D_LIBCPP_ABI_VERSION=Cr -D_LIBCPP_ENABLE_NODISCARD -D_LIBCPP_DEBUG=0 -DCR_LIBCXX_REVISION=79a2e924d96e2fc1e4b937c42efd08898fa472d7 -DCR_SYSROOT_HASH=60354520bb7001d8d0288a2ab82cec7efec9fa3d -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -D_LIBCPP_BUILDING_LIBRARY -I../.. -Igen '-I../../buildtools/third_party/libc++' -fno-delete-null-pointer-checks -fno-ident -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -funwind-tables -fPIC -pthread -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 -ffp-contract=off -fcomplete-member-pointers -m64 -msse3 -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -ffile-compilation-dir=. -no-canonical-prefixes -ftrivial-auto-var-init=pattern -O0 -fno-omit-frame-pointer -gdwarf-4 -g2 -gdwarf-aranges -gsplit-dwarf -ggnu-pubnames -Xclang -fuse-ctor-homing -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-template-as-trivial-member -Xclang -plugin-arg-find-bad-constructs -Xclang use-classify-type -Xclang -plugin-arg-find-bad-constructs -Xclang check-ipc -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -fstrict-aliasing -fPIC -Werror -Wall -Wno-unused-variable '-Wno-c++11-narrowing' -Wno-unused-but-set-variable -Wno-misleading-indentation -Wno-missing-field-initializers -Wno-unused-parameter -Wloop-analysis -Wno-unneeded-internal-declaration -Wenum-compare-conditional -Wno-psabi -Wno-ignored-pragma-optimize -Wno-unqualified-std-cast-call -Wmax-tokens -fvisibility=default -Wno-undefined-bool-conversion -Wno-tautological-undefined-compare '-std=c++17' -Wno-trigraphs -fno-aligned-new '-nostdinc++' '-isystem../../buildtools/third_party/libc++/trunk/include' '-isystem../../buildtools/third_party/libc++abi/trunk/include' --sysroot=../../build/linux/debian_bullseye_amd64-sysroot '-std=c++20' -fexceptions -frtti -c '../../buildtools/third_party/libc++/trunk/src/vector.cpp' -o 'obj/buildtools/third_party/libc++/libc++/vector.o'
[T 11:50:24 92785] Extractor terminated with exit code 1.
[T 11:50:24 92795] Reading configuration file /home/micah/chromium/src/chromium-database/working/tracing/compiler-tracing12366888720025849700.spec...
[T 11:50:24 92795] Initialising tags...
[T 11:50:24 92795] ID set to 0000000000016A7B_0000000000000001 (parent 0000000000016731_0000000000000003)
[T 11:50:24 92795] ==== Candidate to intercept: /bin/sh (canonical: /usr/bin/dash) ====
[T 11:50:24 92796] Reading configuration file /home/micah/chromium/src/chromium-database/working/tracing/compiler-tracing12366888720025849700.spec...
[T 11:50:24 92796] Initialising tags...
[T 11:50:24 92796] ID set to 0000000000016A7C_0000000000000001 (parent 0000000000016A7B_0000000000000001)
[T 11:50:24 92796] ==== Candidate to intercept: ../../third_party/llvm-build/Release+Asserts/bin/clang++ (canonical: /home/micah/chromium/src/third_party/llvm-build/Release+Asserts/bin/clang) ====
[T 11:50:24 92796] === Intercepted call to /home/micah/chromium/src/third_party/llvm-build/Release+Asserts/bin/clang === 
[T 11:50:24 92798] Attempting to switch stdout/stderr to 3...
[T 11:50:24 92798] Reading configuration file /home/micah/chromium/src/chromium-database/working/tracing/compiler-tracing12366888720025849700.spec...
[T 11:50:24 92798] Initialising tags...
[T 11:50:24 92798] ID set to 0000000000016A7E_0000000000000001 (parent 0000000000016A7C_0000000000000001)
[E 11:50:24 92798] CodeQL C/C++ Extractor 2.7.5
[E 11:50:24 92798] Current directory: /home/micah/chromium/src/out/Default
[E 11:50:24 92798] Command: /home/micah/Downloads/codeql/cpp/tools/linux64/extractor --mimic '/home/micah/chromium/src/third_party/llvm-build/Release+Asserts/bin/clang' -MMD -MF 'obj/buildtools/third_party/libc++abi/libc++abi/cxa_aux_runtime.o.d' -DLIBCXXABI_SILENT_TERMINATE -D_LIBCPP_CONSTINIT=constinit -DDCHECK_ALWAYS_ON=1 -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_OZONE=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_GNU_SOURCE '-DCR_CLANG_REVISION="llvmorg-15-init-7040-g50de659a-3"' -DCOMPONENT_BUILD -D_LIBCPP_ABI_UNSTABLE -D_LIBCPP_ABI_VERSION=Cr -D_LIBCPP_ENABLE_NODISCARD -D_LIBCPP_DEBUG=0 -DCR_LIBCXX_REVISION=79a2e924d96e2fc1e4b937c42efd08898fa472d7 -DCR_SYSROOT_HASH=60354520bb7001d8d0288a2ab82cec7efec9fa3d -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -D_LIBCPP_BUILDING_LIBRARY '-I../../buildtools/third_party/libc++/trunk/src' -I../.. -Igen '-I../../buildtools/third_party/libc++' -fno-delete-null-pointer-checks -fno-ident -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -funwind-tables -fPIC -pthread -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 -ffp-contract=off -fcomplete-member-pointers -m64 -msse3 -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -ffile-compilation-dir=. -no-canonical-prefixes -ftrivial-auto-var-init=pattern -O0 -fno-omit-frame-pointer -gdwarf-4 -g2 -gdwarf-aranges -gsplit-dwarf -ggnu-pubnames -Xclang -fuse-ctor-homing -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-template-as-trivial-member -Xclang -plugin-arg-find-bad-constructs -Xclang use-classify-type -Xclang -plugin-arg-find-bad-constructs -Xclang check-ipc -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Werror -Wall -Wno-unused-variable '-Wno-c++11-narrowing' -Wno-unused-but-set-variable -Wno-misleading-indentation -Wno-missing-field-initializers -Wno-unused-parameter -Wloop-analysis -Wno-unneeded-internal-declaration -Wenum-compare-conditional -Wno-psabi -Wno-ignored-pragma-optimize -Wno-unqualified-std-cast-call -Wmax-tokens -fstrict-aliasing -fPIC -fvisibility=default -Wno-undefined-bool-conversion -Wno-tautological-undefined-compare '-std=c++17' -Wno-trigraphs -fno-aligned-new '-nostdinc++' '-isystem../../buildtools/third_party/libc++/trunk/include' '-isystem../../buildtools/third_party/libc++abi/trunk/include' --sysroot=../../build/linux/debian_bullseye_amd64-sysroot -fexceptions -frtti '-std=c++20' -c '../../buildtools/third_party/libc++abi/trunk/src/cxa_aux_runtime.cpp' -o 'obj/buildtools/third_party/libc++abi/libc++abi/cxa_aux_runtime.o'
Warning: Unrecognised command line argument -mllvm
Warning: Unrecognised command line argument -instcombine-lower-dbg-declare=0
Warning: Unrecognised command line argument -mllvm
Warning: Unrecognised command line argument -instcombine-lower-dbg-declare=0
[E 11:50:24 92798] Checking whether C compilation already happened.
[E 11:50:24 92798] Checking for tag c-compilation-happened
[E 11:50:24 92798] Checking CODEQL_TRACER_DB_ID 0000000000016A7C_0000000000000001
[E 11:50:24 92798] Locking DB /home/micah/chromium/src/chromium-database/working/tags.db
[E 11:50:24 92798] Locked DB /home/micah/chromium/src/chromium-database/working/tags.db
[E 11:50:24 92798] Unlocking DB
[E 11:50:24 92798] Unlocked DB
[E 11:50:24 92798] Looks like C compilation didn't already happen.
[E 11:50:24 92798] Checking whether C compilation has been attempted.
[E 11:50:24 92798] Checking for tag c-compilation-attempted
[E 11:50:24 92798] Checking CODEQL_TRACER_DB_ID 0000000000016A7C_0000000000000001
[E 11:50:24 92798] Locking DB /home/micah/chromium/src/chromium-database/working/tags.db
[E 11:50:24 92798] Locked DB /home/micah/chromium/src/chromium-database/working/tags.db
[E 11:50:24 92798] Unlocking DB
[E 11:50:24 92798] Unlocked DB
[E 11:50:24 92798] Marking C compilation as attempted.
[E 11:50:24 92798] Setting tag c-compilation-attempted
[E 11:50:24 92798] Starting from CODEQL_TRACER_DB_ID 0000000000016A7C_0000000000000001
[E 11:50:24 92798] Locking DB /home/micah/chromium/src/chromium-database/working/tags.db
[E 11:50:24 92798] Locked DB /home/micah/chromium/src/chromium-database/working/tags.db
[E 11:50:24 92798] Set tag for0000000000016A7C_0000000000000001
[E 11:50:24 92798] Set tag for0000000000016A7B_0000000000000001
[E 11:50:24 92798] Set tag for0000000000016731_0000000000000003
[E 11:50:24 92798] Set tag for0000000000016731_0000000000000002
[E 11:50:24 92798] Set tag for0000000000016731_0000000000000001
[E 11:50:24 92798] Set tag for0000000000016728_0000000000000004
[E 11:50:24 92798] Set tag for0000000000016728_0000000000000003
[E 11:50:24 92798] Set tag for0000000000016728_0000000000000002
[E 11:50:24 92798] Set tag forroot
[E 11:50:24 92798] Unlocking DB
[E 11:50:24 92798] Unlocked DB
[E 11:50:24 92798] Processed command line: /home/micah/Downloads/codeql/cpp/tools/linux64/extractor --trapfolder /home/micah/chromium/src/chromium-database/trap/cpp --src_archive /home/micah/chromium/src/chromium-database/src --mimic_config /home/micah/chromium/src/chromium-database/working/compiler_mimic_cache/bc9026de7277 --object_filename 'obj/buildtools/third_party/libc++abi/libc++abi/cxa_aux_runtime.o' -w --error_limit 1000 --disable_system_macros --variadic_macros '--c++20' --gcc --gnu_version 40000 --clang -DLIBCXXABI_SILENT_TERMINATE -D_LIBCPP_CONSTINIT=constinit -DDCHECK_ALWAYS_ON=1 -DUSE_UDEV -DUSE_AURA=1 -DUSE_GLIB=1 -DUSE_OZONE=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_GNU_SOURCE '-DCR_CLANG_REVISION="llvmorg-15-init-7040-g50de659a-3"' -DCOMPONENT_BUILD -D_LIBCPP_ABI_UNSTABLE -D_LIBCPP_ABI_VERSION=Cr -D_LIBCPP_ENABLE_NODISCARD -D_LIBCPP_DEBUG=0 -DCR_LIBCXX_REVISION=79a2e924d96e2fc1e4b937c42efd08898fa472d7 -DCR_SYSROOT_HASH=60354520bb7001d8d0288a2ab82cec7efec9fa3d -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -D_LIBCPP_BUILDING_LIBRARY '-I../../buildtools/third_party/libc++/trunk/src' -I../.. -Igen '-I../../buildtools/third_party/libc++' -D__DATE__= -D__TIME__= -D__TIMESTAMP__= --isysroot ../../build/linux/debian_bullseye_amd64-sysroot --exceptions --rtti --sys_include '../../buildtools/third_party/libc++/trunk/include' --sys_include '../../buildtools/third_party/libc++abi/trunk/include' -- '../../buildtools/third_party/libc++abi/trunk/src/cxa_aux_runtime.cpp'
Command-line error: language modes specified are incompatible

[pooneh-nb]

A quick update; I added a few arguments in the process of making ninja files using gn. So instead of running gn gen out/Default, I ran gn gen out/Default --args="is_component_build = false is_debug = false enable_nacl = false symbol_level = 0 blink_symbol_level = 0 v8_symbol_level = 0". Then created the codeql database codeql database create chromium-database --language=cpp --command="autoninja -C out/Default chrome”
The result changes significantly. Now instead of 3k files, I have 85K files but it is still much less than the original number of files in the chromium source code. Here is the comparison:

	Original chormium	Codeql src
Total files in database	847.207	85.758
Total .c, .cc, .cpp, .cxx files	109.927	35.468
Total .h, .hxx, .hpp, .hh files	108.392	49.765

[ducphanduyagentp]

I've recently experienced the same outcome. Is there any update on the situation? I can also share the error logs if it helps.

Edit 1: I switched to CodeQL CLI 2.8.2 and add a few build arguments to chromium (is_debug=false is_component_build=false dcheck_always_on=false) and it seems to be generating the right amount of output, but I still experienced some catastrophic errors. I will try a few more combinations of CodeQL version and build configs to see what works and what doesn't. The first combination that didn't work was v2.10.1 (latest) with default build configuration from gn gen out/Default.

[xeroskim]

Are there any updates on this problem? I'm facing the same issue.

[jketema]

Hi all, the problem with language modes specified are incompatible should be fixed in CodeQL 2.12.2, which will be released in a few weeks time. I'm closing this issue, but feel free to re-open if you spot more language modes specified are incompatible problems. Please open new issues for any other problems related to missing files.