clang12: Using as compiler generates 30K warnings of `-Wcompound-token-split-by-macro`

[jkeenan]

Today I had occasion to try to build perl on FreeBSD-12 with, for the first time clang-12. Using this version of clang introduced over 30,000 new build-time warnings, all of type -Wcompound-token-split-by-macro.

$ uname -mrs
FreeBSD 12.2-STABLE amd64

$ ./perl -Ilib -V:config_args
config_args='-des -Dusedevel -Dusethreads -Dcc=clang12';

$ clang12 --version
clang version 12.0.0
Target: x86_64-portbld-freebsd12.2
Thread model: posix
InstalledDir: /usr/local/llvm12/bin

$ ~/bin/perl/report-build-warnings 1e43c13e1f.freebsd.clang12.maketp.output.txt.gz
File:  1e43c13e1f.freebsd.clang12.maketp.output.txt.gz

  Wcompound-token-split-by-macro           30106
  Wstring-compare                            2

$ zgrep "source =>" 1e43c13e1f.freebsd.clang12.parse.output.txt.gz | sort | uniq -c
  16     source => "../../../hv_func.h",
 144     source => "../../../inline.h",
 208     source => "../../../perl_siphash.h",
  96     source => "../../../sbox32_hash.h",
 272     source => "../../../zaphod32_hash.h",
 122     source => "../../hv_func.h",
1124     source => "../../inline.h",
1586     source => "../../perl_siphash.h",
 732     source => "../../sbox32_hash.h",
2074     source => "../../zaphod32_hash.h",
  36     source => "./const-c.inc",
  12     source => "./const-xs.inc",
  26     source => "./constants.xs",
  86     source => "./hv_func.h",
 774     source => "./inline.h",
  10     source => "./invlist_inline.h",
   6     source => "./numeric.xs",
1118     source => "./perl_siphash.h",
 516     source => "./sbox32_hash.h",
  16     source => "./vutil.c",
  32     source => "./vxs.inc",
1462     source => "./zaphod32_hash.h",
1180     source => "APItest.c",
 506     source => "APItest.xs",
 198     source => "B.c",
 140     source => "B.xs",
  20     source => "Base64.c",
   8     source => "Base64.xs",
  10     source => "Byte.c",
  90     source => "Bzip2.c",
  52     source => "Bzip2.xs",
  10     source => "CN.c",
  10     source => "Call.c",
  16     source => "Call.xs",
  14     source => "Collate.c",
  26     source => "Collate.xs",
  16     source => "Cwd.c",
  30     source => "Cwd.xs",
 170     source => "DB_File.c",
  60     source => "DB_File.xs",
   2     source => "DosGlob.c",
   2     source => "Dumper.c",
  62     source => "Dumper.xs",
  22     source => "DynaLoader.c",
  10     source => "EBCDIC.c",
  42     source => "Encode.c",
 120     source => "Encode.xs",
  20     source => "FastCalc.c",
  26     source => "FastCalc.xs",
  12     source => "Fcntl.xs",
  18     source => "FieldHash.c",
  14     source => "FieldHash.xs",
 132     source => "GDBM_File.c",
  32     source => "GDBM_File.xs",
  10     source => "Glob.c",
  24     source => "Glob.xs",
  84     source => "HiRes.c",
  18     source => "HiRes.xs",
   6     source => "Hostname.xs",
  46     source => "IO.c",
  20     source => "IO.xs",
  10     source => "JP.c",
  10     source => "KR.c",
   2     source => "Langinfo.c",
  66     source => "ListUtil.c",
 320     source => "ListUtil.xs",
   8     source => "MD5.c",
  24     source => "MD5.xs",
  64     source => "NDBM_File.c",
   2     source => "NDBM_File.xs",
  38     source => "Normalize.c",
  20     source => "Normalize.xs",
  24     source => "Opcode.c",
  22     source => "Opcode.xs",
 342     source => "POSIX.c",
  98     source => "POSIX.xs",
  28     source => "Peek.c",
   4     source => "Piece.c",
   4     source => "Piece.xs",
1036     source => "RealPPPort.c",
 696     source => "RealPPPort.xs",
  70     source => "SDBM_File.c",
   2     source => "SDBM_File.xs",
  38     source => "SHA.c",
  60     source => "SHA.xs",
  24     source => "Socket.c",
  46     source => "Socket.xs",
  26     source => "Storable.c",
2012     source => "Storable.xs",
  10     source => "Symbol.c",
  14     source => "SysV.c",
  96     source => "SysV.xs",
  48     source => "Syslog.c",
  10     source => "TW.c",
 264     source => "Typemap.c",
  42     source => "Typemap.xs",
   4     source => "Unicode.c",
  48     source => "Unicode.xs",
  20     source => "Util.c",
  86     source => "Util.xs",
   8     source => "XSUB-redefined-macros.xs",
   4     source => "XSUB-undef-XS_VERSION.xs",
 284     source => "Zlib.c",
  58     source => "Zlib.xs",
  20     source => "attributes.xs",
   8     source => "av.c",
   6     source => "caretx.c",
  62     source => "doio.c",
  42     source => "doop.c",
  42     source => "dump.c",
 134     source => "encoding.xs",
   6     source => "exception.c",
  64     source => "gv.c",
  34     source => "hv.c",
   8     source => "locale.c",
   2     source => "mathoms.c",
  98     source => "mg.c",
   6     source => "module3.c",
  52     source => "mro.xs",
   4     source => "mro_core.c",
  20     source => "numeric.c",
 592     source => "op.c",
  24     source => "pad.c",
 388     source => "perl.c",
  28     source => "perlio.c",
   6     source => "perly.c",
  12     source => "perly.y",
 834     source => "pp.c",
 252     source => "pp_ctl.c",
 366     source => "pp_hot.c",
 264     source => "pp_pack.c",
  30     source => "pp_sort.c",
 494     source => "pp_sys.c",
   2     source => "re.c",
  20     source => "re.xs",
2000     source => "re_comp.c",
 712     source => "re_exec.c",
  12     source => "reentr.c",
1476     source => "regcomp.c",
 266     source => "regexec.c",
   8     source => "scalar.xs",
   6     source => "scope.c",
  52     source => "shared.c",
 354     source => "shared.xs",
 760     source => "sv.c",
  42     source => "threads.c",
 190     source => "threads.xs",
  30     source => "time64.c",
 164     source => "toke.c",
 178     source => "universal.c",
  20     source => "utf8.c",
  58     source => "util.c",
  22     source => "via.xs",

The new warnings occur in pairs like this:

[
  {
    char   => 5,
    group  => "Wcompound-token-split-by-macro",
    line   => 150,
    source => "./zaphod32_hash.h",
    text   => "'(' and '{' tokens introducing statement expression appear in different macro expansion contexts",
  },
  {
    char   => 5,
    group  => "Wcompound-token-split-by-macro",
    line   => 150,
    source => "./zaphod32_hash.h",
    text   => "'}' and ')' tokens terminating statement expression appear in different macro expansion contexts",
  },

Or in the make output:

echo @`sh  cflags "optimize='-O2'" opmini.o`  -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB opmini.c
@clang12 -c -DPERL_CORE -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -std=c89 -O2 -Wall -Werror=pointer-arith -Wextra -Wc++-compat -Wwrite-strings -Werror=declaration-after-statement -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB opmini.c
In file included from op.c:163:
In file included from ./perl.h:4085:
In file included from ./hv.h:659:
In file included from ./hv_func.h:34:
In file included from ./sbox32_hash.h:4:
./zaphod32_hash.h:150:5: warning: '(' and '{' tokens introducing statement expression appear in different macro expansion contexts [-Wcompound-token-split-by-macro]
    ZAPHOD32_SCRAMBLE32(state[0],0x9fade23b);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./zaphod32_hash.h:80:38: note: expanded from macro 'ZAPHOD32_SCRAMBLE32'
#define ZAPHOD32_SCRAMBLE32(v,prime) STMT_START {  \
                                     ^~~~~~~~~~
./perl.h:737:29: note: expanded from macro 'STMT_START'
#   define STMT_START   (void)( /* gcc supports "({ STATEMENTS; })" */
                              ^
./zaphod32_hash.h:150:5: note: '{' token is here
    ZAPHOD32_SCRAMBLE32(state[0],0x9fade23b);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./zaphod32_hash.h:80:49: note: expanded from macro 'ZAPHOD32_SCRAMBLE32'
#define ZAPHOD32_SCRAMBLE32(v,prime) STMT_START {  \
                                                ^
./zaphod32_hash.h:150:5: warning: '}' and ')' tokens terminating statement expression appear in different macro expansion contexts [-Wcompound-token-split-by-macro]
    ZAPHOD32_SCRAMBLE32(state[0],0x9fade23b);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./zaphod32_hash.h:87:41: note: expanded from macro 'ZAPHOD32_SCRAMBLE32'
    v ^= (v>>23);                       \
                                        ^
./zaphod32_hash.h:150:5: note: ')' token is here
    ZAPHOD32_SCRAMBLE32(state[0],0x9fade23b);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./zaphod32_hash.h:88:3: note: expanded from macro 'ZAPHOD32_SCRAMBLE32'
} STMT_END
  ^~~~~~~~
./perl.h:738:21: note: expanded from macro 'STMT_END'
#   define STMT_END     )
                        ^

[Leont]

Sounds like we need to disable that warning? I'm not seeing the use of this warning.

[jkeenan]

One data point: The presence of this warning in our make output is new to clang12. I just installed LLVM/clang 11 on the same machine and built perl with the same config_args. The -Wcompound-token-split-by-macro warnings don't appear in clang11.

$ ~/bin/perl/report-build-warnings 1e43c13e1f.freebsd.clang11.maketp.output.txt.gz 
File:  1e43c13e1f.freebsd.clang11.maketp.output.txt.gz

  Wstring-compare                            2

[jkeenan]

Today I inspected the pull request submitted by @Biswa96 in #18879. I built a perl at commit 8ee57ef on FreeBSD-12 using clang12 as the compiler. The pull request eliminates the vast majority of build-time warnings discussed in this ticket.

$ report-build-warnings 8ee57ef583.freebsd.clang12.threaded.Biswa96.maketp.output.txt.gz
File:  8ee57ef583.freebsd.clang12.threaded.Biswa96.maketp.output.txt.gz

  Wcompound-token-split-by-macro           1804
  Wsign-compare                              2

(The -Wsign-compare warnings are from ListUtil.xs for which I have filed a ticket upstream.)

Bucketing the remaining build-time warnings we get:

$ grep "source =>" clang12.Biswa96.parsed-warnings.pl | grep -v 'ListUtil.xs' |sort | uniq -c
  46     source => "IO.c",
  22     source => "IO.xs",
1036     source => "RealPPPort.c",
 694     source => "RealPPPort.xs",
   6     source => "module3.c",

Of these, IO.c, RealPPPort.c and RealPPPort.xs apper to be generated files. module3.c is a test file underneath dist/Devel-PPPort/. That leaves dist/IO/IO.xs to be addressed.

[jkeenan]

I found that applying the change provided by @Biswa96 for perl.h to Devel-PPPort eliminates the remaining 1,804 -Wcompound-token-split-by-macro clang12 build-time warnings.

$ gitshowf |cat
commit 88e366d42b36d12b0fed022ac5209f0151150740
Author:     James E Keenan <jkeenan@cpan.org>
AuthorDate: Tue Jul 13 00:58:50 2021 +0000
Commit:     James E Keenan <jkeenan@cpan.org>
CommitDate: Tue Jul 13 00:58:50 2021 +0000

    Devel-PPPort: clang shows warnings with gcc brace groups
    
    Applying same change that Biswapriyo Nath provided in commit befee8d7e4
    for perl.h.  This eliminates remaining blead-upstream build-time
    warnings generated by compiling with clang12.

diff --git a/dist/Devel-PPPort/parts/inc/misc b/dist/Devel-PPPort/parts/inc/misc
index 9ec0a04df2..3718dd9b5b 100644
--- a/dist/Devel-PPPort/parts/inc/misc
+++ b/dist/Devel-PPPort/parts/inc/misc
@@ -240,7 +240,9 @@ __UNDEF_NOT_PROVIDED__  PERL_GCC_BRACE_GROUPS_FORBIDDEN
 #  endif
 #endif
 
-#if defined(__GNUC__) && !defined(PERL_GCC_BRACE_GROUPS_FORBIDDEN) && !defined(__cplusplus)
+/* clang shows warnings for tokens introducing statement expression appear
+ * in different macro expansion contexts. */
+#if defined(__GNUC__) && !defined(PERL_GCC_BRACE_GROUPS_FORBIDDEN) && !defined(__cplusplus) && !defined(__clang__)
 #  ifndef PERL_USE_GCC_BRACE_GROUPS
 #    define PERL_USE_GCC_BRACE_GROUPS
 #  endif

I applied this commit to the following branch, which is available for smoke-testing:
smoke-me/jkeenan/Biswa96/fix-clang-mingw

Thank you very much.
Jim Keenan

[jkeenan]

Applying the patch provided by @Biswa96 to both perl.h and Devel-PPPort appears to clear up the build-time warnings generated when using clang12 as the compiler.

Evidence for this comes from smoke-test reports on the smoke-me/jkeenan/Biswa96/fix-clang-mingw branch. Consider these two recent reports from perl@greerga.m-l.org which appear to have the same configuration and have been run on the same machine.

blead
http://perl.develop-help.com/raw/?id=269540

Automated smoke report for branch blead 5.35.2 patch c9d678845a114f97048519e9c564b342b08ccfd2 v5.35.1-106-gc9d678845a
vier.local: Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz (GenuineIntel 3702MHz) (x86_64/1 cpu[8 cores])
    on        linux - 5.3.12-200.fc30.x86_64 [Fedora 34]
    using     ccache clang version Clang 12.0.0 (Fedora 12.0.0-2.fc34)
    smoketime 4 hours 12 minutes (average 1 hour 3 minutes)
    
v5.35.1-106-gc9d678845a  Configuration (common) 
-Accflags="-Werror=declaration-after-statement -g 
-fno-omit-frame-pointer -fsanitize=address  -fno-common 
-fsanitize-blacklist=`pwd`/asan_ignore" 
-Aldflags="-fsanitize=address" -Dcc="ccache clang"


$ zgrep -c -E 'warning:.*-W' logc9d678845a114f97048519e9c564b342b08ccfd2.log.gz 
134467

smoke-me/jkeenan/Biswa96/fix-clang-mingw
http://perl.develop-help.com/raw/?id=269579

Automated smoke report for branch smoke-me/jkeenan/Biswa96/fix-clang-mingw 5.35.2 patch 88e366d42b36d12b0fed022ac5209f0151150740 v5.35.1-112-g88e366d42b
vier.local: Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz (GenuineIntel 3702MHz) (x86_64/1 cpu[8 cores])
    on        linux - 5.3.12-200.fc30.x86_64 [Fedora 34]
    using     ccache clang version Clang 12.0.0 (Fedora 12.0.0-2.fc34)
    smoketime 4 hours 16 minutes (average 1 hour 4 minutes)
    
v5.35.1-112-g88e366d42b  Configuration (common) 
-Accflags="-Werror=declaration-after-statement -g 
-fno-omit-frame-pointer -fsanitize=address  -fno-common 
-fsanitize-blacklist=`pwd`/asan_ignore" 
-Aldflags="-fsanitize=address" -Dcc="ccache clang"

$ zgrep -c -E 'warning:.*-W' log88e366d42b36d12b0fed022ac5209f0151150740.log.gz 
8

[tonycoz]

I suspect the -Wcompound-token-split-by-macro warning is intended to highlight macros that might end up causing errors, and I think it's in general a good warning.

#18879 suggested disabling PERL_GCC_BRACE_GROUPS to prevent those warning, but that also disables many useful -DEBUGGING checks in sv.h.

I think a better alternative would be for STMT_START/STMT_END to not depend on PERL_GCC_BRACE_GROUPS, I don't think using the gcc extension here adds any value, since these are documented to not work as an expression.

Unfortunately Devel::PPPort also includes the same definition, so it also needs to be updated.

See #18984

[khwilliamson]

The commit that fixed this breaks Perl 5.13.0 in Devel::PPPort and needs to be reverted or changed