Dependency compilation improvements

[robdimsdale]

There are some improvements we should make to the way we compile python.

1. Add --enable-optimizations for some versions of Cpython. This flag improves run-time performance by about 10% (according to the internet) at the expense of longer compile times. It is enabled on every common pre-built python binary (e.g. the python distributed by Canonical, the Python Docker images maintained by Docker, etc).
   • It leverages profile-guided optimization - essentially running a set of performance tests during the compilation process to make the resultant compiled binary more efficient.
   • Older versions like 3.7.x have a very large test suite which takes a long time to run (often over an hour). Newer versions have a smaller test suite which takes more like 5-10 minutes to run. We should evaluate this flag for each version of Cpython, and only enable it when it does not make the build take too long.
   • It is possible to manually curate a list of tests to run in the PGO step, which would reduce the build duration for older versions of Cpython, but this seems like a lot of work for us to understand and maintain. I'd like to wait for feedback that older versions of python are slow before doing this work.
   • We have not observed a noticeable increased in file size (e.g. 77MB vs 78MB for Cpython 3.11.0)
2. Use make -j$(nproc) LDFLAGS="-Wl,--strip-all"
   • make -j speeds up build time by a minute or two
   • strip reduces filesize by ~10% with no observed impact on build time
   • we can keep the make install step as-is - it does not benefit from parallelization.
3. Ensure the compilation environment has the necessary development libraries. E.g. on Cpython 3.11.0 we see this output during the compilation step:

   The necessary bits to build these optional modules were not found:
   _bz2                  _curses               _curses_panel
   _lzma                 _tkinter              readline
   

Additionally, we should evaluate the --with-lto flag in conjunction with --enable-optimizations. In principle, link-time optimization (LTO) should improve run-time performance with minimal impact to build duration and a slight increase in file size. I have seen this on other languages like Rust. However, preliminary investigation for Cpython shows that this makes the build much slower (typically 5-10x slower than just using --enable-optimizations for Cpython 3.11.0), and it significantly increases the resultant filesize (we observed about 50% increase for Cpython 3.11.0).

[robdimsdale]

We should also consider using the base stack build image for compilation, but we can split that off from this investigation if we wish.

[jpena-r7]

Great investigation and write-up. Adding my two cents here.

1. I see that you removed --enabled-optimizations here and I can see the benefit of faster builds (5x) at the likely imperceptible cost of performance. I agree that the integration tests became painful after adding support for all stacks, so this seems like a reasonable option to improve that.
   • From my perspective, anyone trying to get good performance out of python is likely going to be using libraries like numpy, which are built on lower level languages.
   • +1 for adding the flag for the compiled binaries for the paketo stacks. Yet another benefit of using paketo stacks.
2. Agreed for this, and that is what is done when compiling from source on non-paketo stacks.
3. I think it's a good idea to have the dependencies installed in the compilation environment, but I don't know about the best way to check for such messages. I worry about the message changing somehow and failing silently. I believe the amazonlinux stack that is used for testing has all of those installed.

I have no experience with link-time-optimization in general, but I suspect smaller image sizes would be valued more highly. I think any users that need that level of control over the python interpreter will likely be using a custom builder and ensuring compilation dependencies are present.

I hope this is helpful.

[robdimsdale]

Using --enable-optimizations in real-life settings

Yeah, I'm sort of the opinion that a user who is running the Python buildpacks against non-paketo stacks is probably sufficiently aware of what they're doing that they can be responsible for adding whatever compilation/configuration flags they need, including --enable-optimizations.

Related to this - Python compilation takes a fairly long time (even with --enable-optimizations removed) so I expect that users running on their own stacks who don't understand that they are compiling from source are probably more likely to file an issue complaining about a slow build rather than slightly slower runtime performance.

I'm hoping to improve the buildpack output to be more clear about when the buildpack is compiling Python from source vs using a pre-compiled binary. I would like to tackle #474 first, though, as I think a more obvious split between pre-compiled and source dependencies is necessary before we start adding more logic based on which one is selected.

Improvements to the pre-compiled build logic

Yup! I'd like Paketo's Python to be as close to the performance of typical third-party distributions (e.g. Ubuntu's distribution of Python), within the constraints of Paketo's other priorities like reproducible builds. See paketo-buildpacks/python#507 for a discussion of the trade-offs of disabling the pycache files in order to facilitate reproducible builds.

If you have any other suggestions for compile-time improvements I'd love to hear them!

Linked/Dependent packages

I agree that validating Python has been compiled against specific libraries could result in brittle tests. I think the best way to validate that we compiled Python against the desired list of dependent libraries is by asserting specific behavior: e.g. can we connect to https? Then openssl was linked correctly. Can we uncompress a bz file? Then the bz library was linked correctly.

That style of testing should be both more useful - the tests are more representative - as well as less brittle - it doesn't rely on specific names of packages. It is more work though 😅

LTO

I'm also leaning towards not compiling with LTO. I'm honestly a little surprised it makes the Python distribution much larger, given that I've had the opposite experience with many Rust applications. But I guess a lot of it is down to the specific code being compiled.

But, before we make that decision, I'd like to do some performance testing of Python compiled with and without LTO. If the run-time performance is much better, we might want to re-evaluate.

[robdimsdale]

This will be partially addressed in #476

[robdimsdale]

This was also partially addressed in #478