bpo-28015: Support LTO build with clang

[serge-sans-paille]

.o generated by clang in LTO mode actually are LLVM bitcode files, which
leads to a few errors during configure/build step:

• add lto flags to the BASECFLAGS instead of CFLAGS, as CFLAGS are used
  to build autoconf test case, and some are not compatible with clang LTO
  (they assume binary in the .o, not bitcode)
• force llvm-ar instead of ar, as ar is not aware of .o files generated
  by clang -flto

https://bugs.python.org/issue28015

[the-knights-who-say-ni]

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA (this might be simply due to a missing "GitHub Name" entry in your b.p.o account settings). This is necessary for legal reasons before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

You can check yourself to see if the CLA has been received.

Thanks again for your contribution, we look forward to reviewing it!

[serge-sans-paille]

CLA just signed, that should be ok now.

I did not commit the autoreconf-generated file, don't know what's the procedure there.

[vstinner]

Python source code includes configure for practical reasons, you must run autoconf and include the modified configure in your PR.

[bedevere-bot]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[vstinner]

It would be convenient if configure fails with an error and an explanation how to fix it if llvm-ar is missing:

AC_PATH_TARGET_TOOL(LLVM_AR, llvm-ar, '', ${llvm_path})
(...)
AR="${LLVM_AR}"

[vstinner]

I tested the 3rd version. It seems like -flto is passed during the compilation of object files and to the final linker step:

$ grep lto Makefile
BASECFLAGS=	 -flto -g -Wno-unused-result -Wsign-compare
CONFIGURE_LDFLAGS=	 -flto -g
CONFIG_ARGS=	 '--with-pydebug' 'CC=clang' '--with-lto'

$ make
...
clang -pthread -fPIC -flto (...)
...
/usr/bin/llvm-ar rcs libpython3.8dm.a Modules/getbuildinfo.o (...) Python/frozen.o
clang -pthread -flto -g  -Xlinker -export-dynamic -o python Programs/python.o libpython3.8dm.a -lpthread -ldl  -lutil -lm   -lm 

[vstinner]

LGTM. It works as expected (see my comment above) and it fix an obvious bug https://bugs.python.org/issue28015#msg327817

[vstinner]

You may add a NEWS entry using the tool "blurb".

@serge-sans-paille signed the CLA, but I have to wait until someone review his signature.

[vstinner]

I will remove the "CLA not signed" label once "Contrib Form Received" will become Yes on https://bugs.python.org/user22883

[miss-islington]

Thanks @serge-sans-paille for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

[bedevere-bot]

GH-10089 is a backport of this pull request to the 3.7 branch.

[vstinner]

@serge-sans-paille: The backport to 3.7 was straighfoward and could be automated, but backport to 3.6 creates a conflict on configure.ac. Do you want to try the backport? I'm not sure if it's worth it to backport the change up to 3.6. It's up to you.

[serge-sans-paille]

@vstinner I think I have the 3.6 patch, should I create a new PR on the 3.6 branch?

[serge-sans-paille]

Finally, there a re many complex interactions with RANLIB and AR, dropping the backport.

[vstinner]

Finally, there a re many complex interactions with RANLIB and AR, dropping the backport.

The Python 3.6 backport? It's too complex? It's ok to only fix Python 3.7 and newer. (I just need your confirmation to close the issue.)

[serge-sans-paille]

Yeah, the 3.6 backport would require some more thoughts, better drop it and close the issue. Thanks!

[stratakis]

The 3.6 backport works fine if https://bugs.python.org/issue31625 is applied for 3.6 as well.