Tons of include file notices when building with Chinese version of Visual Studio 2010

[sfcheng]

When building chromium with ninja with Chinese version of Visual Studio 2010, the console window outputs tons of include file notices and thus greatly slows down the building process. The notice is like:

注意： 包含文件：include file path

The English equivalent is：
Note: including file: .....

Perhaps ninja is stripping the include notices based on the English words only.

[sgraham]

On an English install "Note: including file:%s%s\n" appears as resource in the string table in VC\bin\1033\clui.dll.

1033 is the locale identifier for "English (United States)".

I'm not aware of any command line option to force cl into 1033 locale unfortunately. I assume if multiple locales are installed it uses system settings to determine which to use.

So, I guess we'd have to have add various languages to the prefix search in the /showIncludes parser. :/

[evmar]

I think cmcldeps (the CMake parser of this output) uses a regex, something like "[^:]+: [^:]+: (.*)", to grab all output lines that look like showincludes output. I haven't looked at the code too hard because I'd eventually like to implement something like that and I don't want to violate any copyrights. :)

The tricky part is not confusing showincludes output with warnings. sfcheng, could you paste the Chinese output of Visual Studio cl.exe when showing a warning or error message?

[sgraham]

It looks like that : is not ascii 58, so that might add a wrinkle. Maybe the line number "(\d+)" that will be in errors could be useful signal.

[okuoku]

I'm not aware of any command line option to force cl into 1033 locale unfortunately.

There is no (clean) way to do this, unfortunately. Foreign language versions of VS would have other locale resources in different number(e.g. 1041 for JA).
What we learned: always install EN version of VS, then install language pack if needed :(

But fortunately, "error Cnnnn" and "warning Cnnnn" never get localized. So we can use them as key. But as @sgraham said, line numbers looks like more promising because it would also allow filtering out 'note:' output.

I'm not sure whether or not : are not ascii 58. In Japanese version, these are certainly ascii 58.

FWIW, Japanese output would look like this:

C:\cygwin\home\oku>type main.c
#include <stdio.h>
int nah(void){}; /* Trigger "function must return a value */
main(){return nah();}

C:\cygwin\home\oku>cl /showIncludes main.c
Microsoft(R) C/C++ Optimizing Compiler Version 16.00.40219.01 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

main.c
メモ: インクルード ファイル:  C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\INCLUDE\stdio.h
メモ: インクルード ファイル:   C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\INCLUDE\crtdefs.h
メモ: インクルード ファイル:    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\INCLUDE\sal.h
メモ: インクルード ファイル:     c:\program files (x86)\microsoft visual studio10.0\vc\include\codeanalysis\sourceannotations.h
メモ: インクルード ファイル:    C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\INCLUDE\vadefs.h
メモ: インクルード ファイル:   C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\INCLUDE\swprintf.inl
c:\cygwin\home\oku\main.c(2) : warning C4716: 'nah' : 値を返さなければいけません

[nico]

Related art: https://bugzilla.mozilla.org/show_bug.cgi?id=587372 (approach there: read prefix from an env var, have an autoconf check to verify /showIncludes parsing works. Not great.)

[nico]

Idea similar to the mozilla bug: There could be a toplevel var msvs_includes_prefix, and generators could write that by compiling a dummy #include "knownheader.h" file with /showIncludes and writing whatever is in front of "knownheader.h" in the cl.exe output into that toplevel var. Ninja would then use msvs_includes_prefix as the /showIncludes prefix.

[syntheticpp]

While CMake configurures, the include prefix is read from one dummy build,
https://github.com/Kitware/CMake/blob/master/Modules/CMakeClDeps.cmake
where a regex is used, later on the prefix is passed as argument to the tool,
https://github.com/Kitware/CMake/blob/master/Source/cmcldeps.cxx
and std::string::substr is used for processing the cl.exe output.

I assume ninja also needs to accept an additional (global) argument.
(like nico suggested)

CMake also uses cmcldeps to generate dependencies for .rc files by first "compiling" the .rc file with cl which generates the dependency file, and then processes it with the rc tool.
Not sure if or how this could be integrated into ninja.

[syntheticpp]

#665

Does this work with non-Ascii prefixes?

[nico]

#665 is merged. We might have to iterate on encoding issues a bit still though, so leaving this open until this is verified working.

[fbridault]

With a french local, ninja 1.8.2 and CMake 3.10.2, this still happens...

[nico]

#665 added msvc_deps_prefix. Does cmake set that? @syntheticpp @mathstuf

[mathstuf]

@nico I see code in CMake which references it.

[dtaralla]

Still happening on Visual Studio Community 15.9.7...

[DrFrankenstein]

For the record, this is still happening to me with the following configuration:
CMake 3.14
Ninja 1.8.2 (the one that ships with Visual Studio 2019)
French locale.

EDIT: Better workaround: set VSLANG=1033 in the environment to force CL to output English messages.

Old workaround:
And for those who also hit this issue, my workaround was to comment out the following line in $CMAKE_PATH\share\cmake-3.14\Modules\Platform\Windows-MSVC.cmake:
#set(CMAKE_NINJA_DEPTYPE_${lang} msvc) (line 368 in mine)

This unfortunately causes CMake to generate deps = gcc instead of juste removing the deps line, but that hasn't seemed to break my builds. YMMV. This is a workaround.

[mathstuf]

Setting deps = gcc is probably benign without setting depfile as well.

[DrFrankenstein]

I'll give it a shot this week!

[DrFrankenstein]

Unfortunately, that didn't fix it.

I built ninja from that branch, then used that version of ninja to build itself again, and it still leaked the include messages to the terminal.

[DrFrankenstein]

Ha! I was actually going to look in there tomorrow. Seems like you caught it before I had a chance to.

I just shut my computer down for the night. I'll get back to you tomorrow!

[DrFrankenstein]

It's worth noting, though, deps_prefix should contain the localized string as set in the rules.ninja file (usually detected and set by CMake). It only uses the hard-coded one if it's absent.

I suspect the logic just after it might be the actual culprit. But as I said, I'll do a proper investigation/debugging session tomorrow.

[DrFrankenstein]

The encodings don't match. deps_prefix is in Latin-1 (where the NBSP before the colons is 0xA0), and line is in CP437 for some reason (NBSP = 0xFF).

I think that CL itself is outputting CP437, but the CMake-generated rules.ninja is in Latin-1. I'm guessing that some conversion occured on the CMake side, but that'll require more digging.

EDIT: It seems like CL will output in whatever the console's codepage is. (Source 1, Source 2). I'm not sure how we can force it to be something else.

Perhaps we can bring the two together by converting them both to a common encoding such as UTF-8 (or whatever Ninja prefers to use), e.g. by calling MultiByteToWideChar(CP_OEMCP, ...) on the CL output, and MultiByteToWideChar(1252, ...) on the string that comes from rules.ninja.

[DrFrankenstein]

Thinking back on this... this might be CMake's fault. On Windows, the execute_process command seems to convert the output of the command to UTF-8 internally (and accepts an optional ENCODING parameter to specify the output's encoding). It thus writes it back in UTF-8 in the rules.ninja file (where NBSP is 0xA0 and not 0xFF).

I tried changing CMAKE_DETERMINE_MSVC_SHOWINCLUDES_PREFIX to use ENCODING NONE (perform no conversion), but it seemed to break all sorts of things in CMake.

So the question I'm having now is... should ninja's msvc_deps_prefix be a bitwise match of the compiler's output, or should it be in whatever encoding the file is expected to be, in which case it would be Ninja's job to do the proper conversions from the compiler output?

[mathstuf]

@bradking Thoughts on the encoding and prefix detection here?

[evmar]

Historically ninja has been encoding agnostic (as long as the encoding used the same byte as ASCII for '/'). However, Windows might make that difficult.

[bradking]

Ninja's CLParser::FilterShowIncludes is using memcmp to compare msvc_deps_prefix to lines in MSVC's output so indeed the value needs to be a bitwise match. CMake may need some work to preserve that. CMake currently converts to UTF-8 internally so perhaps all that is missing is converting back to the codepage's encoding when writing the value to build.ninja.

IIRC, MSVC's output encoding can be affected by environment variables and/or flags. That means we may end up with the compiler output in a different encoding than the codepage in which Ninja is operating and using to interpret strings in build.ninja. Such cases may require extra support from Ninja to handle but further investigation would be needed.

[DrFrankenstein]

I couldn't find any environment variable affecting the codepage used by CL. I think it just uses the codepage associated with the process (which is based on the system's regional settings, or by the console settings if the process is running in one).

However, there is an environment variable that sets the language used by CL, VSLANG, which can be useful as a workaround for users affected by this bug. Setting VSLANG=1033 before generating the ninja files will prevent the bug from happening.

[evmar]

Just to restate my above comment in different words: Ninja treats its input files as (encodingless) bytes, and does encoding-ignorant byte comparisons of strings, to attempt to evade these issues. You need the bytes that appear in the build.ninja file to match the bytes that ninja reads from the process stdout, but ninja doesn't care about encodings.

[evilenzo]

Similar issue on last Visual Studio Preview 2022 17.2 with ninja 1.10.2 in CLion 2021.3. Tons of "Note including file: foo.h" even without /showIncludes option set. So annoying

[cpsauer]

Hey all! Came across this while writing some Bazel tooling. After some experimenting, we ended up matching all the languages rather than using VSLANG=1033, because the latter doesn't work if the user doesn't have the English language pack installed. (Seems to work otherwise, though.)

[GuoZiyangTYUST]

A solution not very good:
use Clang rather than MSVC

such "Note include" output by MSVC tool chain,just change it.

用了clang就不能用.nativs了。。。。。。
改了一下源码，提供了二进制版本
https://github.com/GuoZiyangTYUST/ninja/releases/tag/%E5%8E%BB%E9%99%A4%E5%86%97%E4%BD%99%E8%BE%93%E5%87%BA

[hez2010]

Hit this issue as well, so we just switched to msbuild from ninja and the issue was gone. Unexpectedly found that msbuild has been far faster than before :D

[kangjianbin]

I guess the bug is in cmake.

CMake can detect the prefix correctly, and stores it in its variable 'CMAKE_CL_SHOWINCLUDES_PREFIX' (or CMAKE_CXX_CL_SHOWINCLUDES_PREFIX). Note that this prefix is not utf-8 encoding. For example, in my locale it is encoded in 'GB18030'.
However, when creating rules.ninja, cmake converts this prefix to utf-8 and saves the converted data to msvc_deps_prefix.

During build, msvc_deps_prefix (utf-8 encoding) cannot match MSVC's output (gb18030 enconding, et al), so Ninja reports lots of include files.

I switched to cmake-3.20, which doesn't convert the prefix to utf-8, and everything works well.

[mathstuf]

I switched to cmake-3.20, which doesn't convert the prefix to utf-8, and everything works well.

So this can be closed then?

[bradking]

A relevant change in CMake is MR 5089, first included in CMake 3.19. It switched to writing the msvc_deps_prefix as a raw byte sequence regardless of the encoding of build.ninja, since Ninja only compares raw byte sequences to look for such lines in the compiler output.

The best path forward for Ninja upstream is #1812/#2200, which add support for using cl's new equivalent to a gcc depfile.

[kangjianbin]

I switched to cmake-3.20, which doesn't convert the prefix to utf-8, and everything works well.

So this can be closed then?

No as it still exists in CMake 3.24. I checked MR 5089, looks like the issue was introduced again in MR 5860 . When Ninja version is larger than 1.11, CMake assumes Ninja supports UTF-8; so it writes rules.ninja with UTF-8.

[bradking]

@kangjianbin thanks. Please open a CMake upstream issue for that. Include details about the actual encoding of the byte sequence that you're observing.

[kangjianbin]

Created issue https://gitlab.kitware.com/cmake/cmake/-/issues/24068

[bradking]

Please try CMake 3.25.

[kangjianbin]

CMake 3.25 works for me. Thanks.

[fcharlie]

When I update cmake to 3.25 this issue doesn't seem to be resolved.

cmake: 3.25.0
ninja: 1.11.1
chcp: 936

Whether you run cmake --build . Or ninja all, the result is the same. Also, modifying the code page to 54936 has no effect.

Enable Beta: Use Unicode UTF-8 for worldwide language support and restart Windows:

I suggest that cmake/ninja should proactively avoid annoying problems caused by MSVC's multilingual mechanism if it wants to read this information from the standard output of 'cl.exe'.

[kangjianbin]

Enable Beta: Use Unicode UTF-8 for worldwide language support and restart Windows:

Did cmake detect the prefix correctly?

[waruqi]

We can run cl.exe /nologo /showIncludes /c /Foxx xxx.c to try compiling a empty file, then detect the note include prefix automatically.

see https://github.com/xmake-io/xmake/blob/ec1bedd448b5e50aab8cd83792fb7e855b1fa680/xmake/modules/private/tools/cl/parse_include.lua#L30

https://github.com/xmake-io/xmake/blob/ec1bedd448b5e50aab8cd83792fb7e855b1fa680/xmake/plugins/project/ninja/build_ninja.lua#L179

and we need also change build.ninja encoding (utf8) to GBK or ....

function probe_include_note_from_cl()
    local key = "cldeps.parse_include.note"
    local note = detectcache:get(key)
    if not note then
        local cl = find_tool("cl")
        if cl then
            local projectdir = os.tmpfile() .. ".cldeps"
            local sourcefile = path.join(projectdir, "main.c")
            local headerfile = path.join(projectdir, "foo.h")
            local objectfile = sourcefile .. ".obj"
            local outdata = try { function()
                local runenvs = toolchain.load("msvc"):runenvs()
                local argv = {"-nologo", "-showIncludes", "-c", "-Fo" .. objectfile, sourcefile}
                io.writefile(headerfile, "\n")
                io.writefile(sourcefile, [[
                    #include "foo.h"
                    int main (int argc, char** argv) {
                        return 0;
                    }
                ]])
                return vstool.iorunv(cl.program, argv, {envs = runenvs, curdir = projectdir})
            end}
            if outdata then
                for _, line in ipairs(outdata:split('\n', {plain = true})) do
                    note = line:match("^(.-:.-: )")
                    if note then
                        break
                    end
                end
            end
            os.tryrm(projectdir)
        end
        detectcache:set(key, note)
        detectcache:save()
    end
    return note
end

function _add_rules_for_compiler(ninjafile)
    ninjafile:print("# rules for compiler")
    if is_plat("windows") then
        -- @see https://github.com/ninja-build/ninja/issues/613
        local note_include = parse_include.probe_include_note_from_cl()
        if not note_include then
            note_include = "Note: including file:"
        end
        ninjafile:print("msvc_deps_prefix = %s", note_include:trim())
    end

BTW, if /sourceDependencies can be supported, it will be better. waiting for #1806

[cpsauer]

If useful: I think that the last time I checked out /sourceDependencies (and friends) they were only for module dependencies, rather than the normal header inclusion you'd need for this.

Creating a test to extract the include note is very clever :)

[mathstuf]

I think that the last time I checked out /sourceDependencies (and friends) they were only for module dependencies, rather than the normal header inclusion you'd need for this.

Early revisions of the P1689 format included this information but it was excluded in order to keep the initial revisions focused and less cluttered.

[dengchcs]

I'm using Visual Studio 2019 with Chinese locale, and solved this issue simply by upgrading my cmake version to 3.26.

将 cmake 版本升级到 3.26 即可

[eli-schwartz]

We can run cl.exe /nologo /showIncludes /c /Foxx xxx.c to try compiling a empty file, then detect the note include prefix automatically.

Certainly, meson does this as well: https://github.com/mesonbuild/meson/blob/ef5da765af7c5687228ed6e77a19d20bc474c4a4/mesonbuild/backend/ninjabackend.py#L526-L573

So does cmake: https://gitlab.kitware.com/cmake/cmake/-/blob/1ec4c6579117634ddb20962cdb63313dbf79887e/Modules/CMakeDetermineCompilerId.cmake#L1131

Both use regexes a lot more complicated than yours, by the way -- I don't know whether that's necessary but it's plausible that you will sometimes fail to find the right info.

[Nosenzor]

If I understand it correctly ninja generator call MSVC compiler with /showIncludes (/showIncludes) si IMO it's ninja to offer an option to hide it.
I'd like to disable those message as they are really polluting the output.
As a sidenote, if I use a different generator than ninja with the same compiler and same Cmake I don't have all of these prints.

[eli-schwartz]

IMO it's ninja to offer an option to hide it.

Ninja does since about five years ago, it's not clear why the ticket is still open.

As a sidenote, if I use a different generator than ninja with the same compiler and same Cmake I don't have all of these prints.

Probably the Make backend doesn't support showIncludes at all, so you don't get reliable builds. Meanwhile, Visual Studio "cheats" since it's responsible for emitting the translated messages and can always tell exactly what part it is.

Unfortunately everyone other than Microsoft needs to work around Microsoft's non-interoperable design of translating machine-readable messages into the user's prose language.

[waruqi]

We can run cl.exe /nologo /showIncludes /c /Foxx xxx.c to try compiling a empty file, then detect the note include prefix automatically.

Certainly, meson does this as well: https://github.com/mesonbuild/meson/blob/ef5da765af7c5687228ed6e77a19d20bc474c4a4/mesonbuild/backend/ninjabackend.py#L526-L573

So does cmake: https://gitlab.kitware.com/cmake/cmake/-/blob/1ec4c6579117634ddb20962cdb63313dbf79887e/Modules/CMakeDetermineCompilerId.cmake#L1131

Both use regexes a lot more complicated than yours, by the way -- I don't know whether that's necessary but it's plausible that you will sometimes fail to find the right info.

I see that cmake detects it automatically, but I use cmake 3.24.2 and it still doesn't work. Is there any special configuration I need to do to turn it on?

[JKwarn]

I'm using Visual Studio 2019 with Chinese locale, and solved this issue simply by upgrading my cmake version to 3.26.

将 cmake 版本升级到 3.26 即可

CMake 3.26 works for me.
升级到3.26对我来说是有用的。

OS: Windows 11 22H2
QT Creator: 10.0.1

[jhasse]

Seems to be fixed. Please reopen if it still applies.

[Ri0n]

QtCreator comes with CMake 3.27.7 and it's reproducible there.

[mathstuf]

Can you please report back to CMake's issue tracker there with more details?