General issue (No source was seen and extracted)

[FaiqueAli]

Hi,
i am trying to use codeql cli in my cpp project and trying to create a database file in a docker container.
but I am getting this issue
No source code was seen and extracted to C:\xxx.
This can occur if the specified build commands failed to compile or process any code.

• For codebases written in Go, JavaScript, TypeScript, and Python, do not specify
  an explicit --command.
• For other languages, the --command must specify a "clean" build which compiles
  all the source code files without reusing existing build artefacts.

I am using the GNU MinGW compiler to compile my code.
the commands are using the script(.bat) file

cmake -G "MSYS Makefiles" ../
make

waiting for good response :)

[aibaars]

The CodeQL C/C++ analyser tracks which source files are compiled during the build process. The error message says that CodeQL somehow did not "see" any source files. This can have various causes:

• CodeQL did not detect any compiler commands (such as gcc.exe)
• CodeQL failed to parse the source files (or its included files)
• Source files were found and parsed but they were not in the expected source location.

Could you have a look at codeql-database.yml in the database folder? There should be a property named sourceLocationPrefix.

Could you have a look in the src.zip file in the database folder? This file contains copies of the source files that CodeQL analysed. If there are files but the folder structure does not match the sourceLocationPrefix, then you could try to adjust the sourceLocationPrefix (-s, --source-root flag to codeql database create).

If there are few (or no) files in the zip file then CodeQL likely failed to detect the compiler calls. In this case you best have a look at the log/build-tracer.log file in the database folder.

[FaiqueAli]

thank you for the valuable information,
after analyzing the > codeql-database.yml file I found the property > sourceLocationPrefix with the correct codebase path (c:/xxx), but I did not find any src.zip file in the database folder (c:/xxx/codeqldb).

any idea?

[aibaars]

thank you for the valuable information, after analyzing the > codeql-database.yml file I found the property > sourceLocationPrefix with the correct codebase path (c:/xxx), but I did not find any src.zip file in the database folder (c:/xxx/codeqldb).

any idea?

Apologies, the src.zip is only created when there were no errors. There should, however, be a c:/xxx/codeqldb/src folder.

[FaiqueAli]

This is the current structure of the database folder

[aibaars]

Ok, in that case we should have a look in the log folder for a file named build-tracer.log . Hopefully that file contains an explanation of why things did not work at all.

One thing to check; did you run make clean or checked out a fresh copy of the repository? A common pitfall is that make won't do anything if the code has already been built, and as a result CodeQL won't "see" any compilation steps.

[FaiqueAli]

I am running all the steps in docker container in the interactive mode.
I have seen build-tracer.log file, but its too long and have no idea what values should I check..
I am going to clean and give a try one more time an let you know the outcomes..

[FaiqueAli]

since I am working in docker so it already has a copy of uncompiled code.
however, I also tried with make clean option and got the same result (No source code was seen...).
getting a look at the build-tracer.log file got nothing, it's just showing some repetition of steps, like in the image below..

[aibaars]

The build-tracer.log file tends to be very large and repetitive and it can be a bit challenging to find the interesting parts. It would be good to look for lines like Candidate to intercept: ......\gcc.exe . For every compiler call there should also be a line containing \extractor.exe with a ton of command line arguments. Lines containing error: and catastrophic error are also interesting.

If you are comfortable sharing the contents of the build-tracer.log file (please zip it first) then you can attach it to the issue. The log file should not contain any source code, but it does contain file paths of commands and source files.

[FaiqueAli]

Please have a review on this attached .zip
it also contains the script I am providing to database create command.
rootfolder.zip

[aibaars]

I found the relevant bits of the build-tracer.log . I think CodeQL could not understand the output of cc.exe --version and c++.exe --version . I'll pass it on to the team, hopefully there is an easy fix.

[T 10:02:13 1904] ==== Candidate to intercept: C:\msys64\mingw64\bin\cc.exe (1556) ====
[T 10:02:13 1904] Lua: === Intercepted call to c:\msys64\mingw64\bin\cc.exe ===
[T 10:02:13 1904] Executing the following tracer actions:
[T 10:02:13 1904] Tracer actions:
[T 10:02:13 1904] pre_invocations(0)
[T 10:02:13 1904] post_invocations(1)
[T 10:02:13 1904] invocation: C:\codeql\cpp\tools\win64\extractor.exe, args: --mimic "c:\msys64\mingw64\bin\cc.exe" --version
[T 10:02:13 1904] trace_languages(1): [cpp]
[E 10:02:13 1076] CodeQL C/C++ Extractor 2.10.3
[E 10:02:13 1076] Current directory: C:\xxx\build
[E 10:02:13 1076] Command: C:\codeql\cpp\tools\win64\extractor.exe --mimic c:\msys64\mingw64\bin\cc.exe --version
Warning: c:\msys64\mingw64\bin\cc.exe does not look like a compiler. If it is infact a compiler, please contact GitHub.
[E 10:02:14 1076] Not a compiler; exiting. Reason: --version stdout was:
cc.exe (Rev1, Built by MSYS2 project) 12.2.0

Copyright (C) 2022 Free Software Foundation, Inc.

This is free software; see the source for copying conditions.  There is NO

warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.




--version stderr was:

process exited with code 0

and

[T 10:02:14 1904] Lua: === Intercepted call to c:\msys64\mingw64\bin\c++.exe ===
[T 10:02:14 1904] Executing the following tracer actions:
[T 10:02:14 1904] Tracer actions:
[T 10:02:14 1904] pre_invocations(0)
[T 10:02:14 1904] post_invocations(1)
[T 10:02:14 1904] invocation: C:\codeql\cpp\tools\win64\extractor.exe, args: --mimic "c:\msys64\mingw64\bin\c++.exe" --version
[T 10:02:14 1904] trace_languages(1): [cpp]
[E 10:02:14 1880] CodeQL C/C++ Extractor 2.10.3
[E 10:02:14 1880] Current directory: C:\xxx\build
[E 10:02:14 1880] Command: C:\codeql\cpp\tools\win64\extractor.exe --mimic c:\msys64\mingw64\bin\c++.exe --version
Warning: c:\msys64\mingw64\bin\c++.exe does not look like a compiler. If it is infact a compiler, please contact GitHub.
[E 10:02:15 1880] Not a compiler; exiting. Reason: --version stdout was:
c++.exe (Rev1, Built by MSYS2 project) 12.2.0

Copyright (C) 2022 Free Software Foundation, Inc.

This is free software; see the source for copying conditions.  There is NO

warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.




--version stderr was:

process exited with code 0

[FaiqueAli]

Thank you very much for the identification, I hope we will have some good fix.
Now waiting for the fix, in order to include codeql in my workflow.

second, I need one clarification,
In my docker file, first I am compiling my code without codeql which is a normal process before codeql integration into my workflow.
Now adding codeql in my workflow, I need to compile my code again which means I need to compile my code two times.
is there any way in which I can omit the second-time compilation?
because the codebase compilation takes time (10min to 25 min) and increases performance issues on the server.

Thanks :)

[aibaars]

Now adding codeql in my workflow, I need to compile my code again which means I need to compile my code two times. is there any way in which I can omit the second-time compilation? because the codebase compilation takes time (10min to 25 min) and increases performance issues on the server.

You could skip the first compilation and compile your code only once with CodeQL. CodeQL shouldn't change the compilation process.

If you're using the https://github.com/github/codeql-action them you can add a codeql-action/init step before your normal build command, and add a codeql-action/analyze step right after the build step.

You can do something similar with the codeql CLI using a feature called indirect build tracing. See also: https://codeql.github.com/docs/codeql-cli/creating-codeql-databases/#using-indirect-build-tracing

[FaiqueAli]

Hi aibaars,
any information about the fix?
I am waiting for your response..

Thanks:)