[FR] Consider switching to `nanobind` Python bindings

[nicholasjng]

Is your feature request related to a problem? Please describe.
Not a problem, more an idea: For Python >=3.8 bindings using C++17, the nanobind project (from the same hand as pybind11) promises binary size reductions and runtime performance increases (~40% in their benchmarks) over pybind11 for frequently re-called functions in a loop.

For a benchmark library, runtime performance benefits like these could be interesting - especially since the quoted runtime performance increase for "calling functions repeatedly in a loop" fits the bill exactly in this case.

Describe the solution you'd like
Implement the GBM Python bindings in nanobind instead, measure the performance / binding size differences, decide where to go from there after interpreting the results.

Describe alternatives you've considered
Staying with the current thing.

Additional context
I want to do this as a personal experiment, to learn how to implement C++ Python bindings. If something comes out of it other than the nanobind-specific results (better/any docstrings, type hints etc), they could be added to the existing bindings, too.

Not sure how stable nanobind is right now, but since it has pybind11 as its direct predecessor project, it might already work sufficiently for this scope.

[nicholasjng]

So I ported over GBM to nanobind, since the necessary tools used are now fully available in nanobind. The results (all on macOS 12.6.2, Apple M1 arm64, Python 3.10.9):

Binding sizes

Measured as wheel size on disk.

• using pybind11: 252KB
• using nanobind: 189KB (-25%)

Compile / wheel build times

Measured with time python setup.py bdist_wheel, with a preceding bazel clean (so that build caches are removed, but dependency downloads stay, and thus do not influence the time).

pybind11: python setup.py bdist_wheel 0.18s user 0.06s system 3% cpu 7.145 total
nanobind: python setup.py bdist_wheel 0.17s user 0.06s system 4% cpu 5.063 total (-29%)

Example script timings / iterations

Created with python bindings/python/google_benchmark/example.py.

With pybind11:

➜ python bindings/python/google_benchmark/example.py
Unable to determine clock rate from sysctl: hw.cpufrequency: No such file or directory
This does not affect benchmark measurements, only the metadata output.
2023-01-10T20:41:19+01:00
Running bindings/python/google_benchmark/example.py
Run on (10 X 24.1213 MHz CPU s)
CPU Caches:
  L1 Data 64 KiB
  L1 Instruction 128 KiB
  L2 Unified 4096 KiB (x10)
Load Average: 2.19, 2.25, 1.81
------------------------------------------------------------------------------
Benchmark                                    Time             CPU   Iterations
------------------------------------------------------------------------------
empty                                      133 ns          133 ns      4902682
sum_million                            8235747 ns      8235750 ns           84
pause_timing                              4968 ns         4981 ns       140006
skipped                             ERROR OCCURRED: 'some error'
manual_timing                         12080841 ns        39046 ns         1000
custom_counters                            144 ns          144 ns      4821568 foo=626.804k foo_avg=626.804k foo_avg_rate=905.901k/s foo_inv_rate=1.10387us foo_rate=905.901k/s
with_options/process_time/real_time    8239830 ns      8239835 ns           85
sum_million_microseconds                  8241 us         8241 us           84
passing_argument/100                       690 ns          690 ns      1006803
passing_argument/1000                     7323 ns         7323 ns        94926
using_range/8                              398 ns          398 ns      1756561
using_range/64                             582 ns          582 ns      1191712
using_range/512                           3299 ns         3299 ns       212006
using_range/4096                         32873 ns        32873 ns        21264
using_range/8192                         66651 ns        66651 ns        10448
computing_complexity/1024                 7523 ns         7523 ns        92412
computing_complexity/2048                15973 ns        15973 ns        43655
computing_complexity/4096                32868 ns        32868 ns        21251
computing_complexity/8192                66981 ns        66981 ns        10421
computing_complexity/16384              134238 ns       134238 ns         5190
computing_complexity/32768              269351 ns       269351 ns         2596
computing_complexity/65536              539761 ns       539761 ns         1287
computing_complexity/131072            1083345 ns      1083096 ns          647
computing_complexity/262144            2161234 ns      2161232 ns          323
computing_complexity_BigO                 8.25 N          8.25 N
computing_complexity_RMS                     0 %             0 %

With nanobind:

➜ python bindings/python/google_benchmark/example.py
Unable to determine clock rate from sysctl: hw.cpufrequency: No such file or directory
This does not affect benchmark measurements, only the metadata output.
2023-01-10T21:52:49+01:00
Running bindings/python/google_benchmark/example.py
Run on (10 X 24.1209 MHz CPU s)
CPU Caches:
  L1 Data 64 KiB
  L1 Instruction 128 KiB
  L2 Unified 4096 KiB (x10)
Load Average: 1.34, 1.58, 1.71
------------------------------------------------------------------------------
Benchmark                                    Time             CPU   Iterations
------------------------------------------------------------------------------
empty                                     21.5 ns         21.5 ns     31814711
sum_million                            8234480 ns      8234476 ns           84
pause_timing                              4704 ns         4705 ns       149003
skipped                             ERROR OCCURRED: 'some error'
manual_timing                         12312186 ns        13633 ns         1000
custom_counters                           31.7 ns         31.7 ns     22059055 foo=2.86768M foo_avg=2.86768M foo_avg_rate=4.10243M/s foo_inv_rate=243.758ns foo_rate=4.10243M/s
with_options/process_time/real_time    8233550 ns      8233553 ns           85
sum_million_microseconds                  8234 us         8234 us           85
passing_argument/100                       472 ns          472 ns      1482401
passing_argument/1000                     7095 ns         7095 ns        98082
using_range/8                              184 ns          184 ns      3801992
using_range/64                             364 ns          364 ns      1921282
using_range/512                           3076 ns         3076 ns       227480
using_range/4096                         32590 ns        32590 ns        21464
using_range/8192                         66370 ns        66370 ns        10494
computing_complexity/1024                 7293 ns         7293 ns        95308
computing_complexity/2048                15733 ns        15733 ns        44411
computing_complexity/4096                32334 ns        32334 ns        21628
computing_complexity/8192                66353 ns        66353 ns        10521
computing_complexity/16384              132702 ns       132701 ns         5258
computing_complexity/32768              268735 ns       268735 ns         2603
computing_complexity/65536              534400 ns       534398 ns         1301
computing_complexity/131072            1079196 ns      1079192 ns          647
computing_complexity/262144            2157401 ns      2157401 ns          324
computing_complexity_BigO                 8.23 N          8.23 N
computing_complexity_RMS                     0 %             0 %

Script runtime is about the same in both cases, about 44 seconds.

Interpretation

Raw times are changed, general scaling behaviors are not (see the Big O value at the end, or the scaling of using_range. This is good news as a sanity check of the implementation.

Rather than getting iteration improvements as some percentage of the original, it seems that we obtain a ~100-200ns reduction in both wall and CPU timings across the board (but as much as ~4000ns for longer benchmarks like computing_complexity/262144), possibly due to nanobind's reduction in memory allocations.

For smaller benchmarks (e.g. empty, custom_counters or using_range/8), this translates to drastically higher iteration count. So as a rule of thumb, the smaller the benchmark, the bigger the gain in iterations, and (hopefully) thus statistical accuracy. On the contrary, the longer the benchmark, the more accurate the base timings are, likely due to better memory management on the bindings side.

All in all not a quantum leap, but still a pretty big win for just writing a BUILD file and changing py:: to nb:: a dozen times. I would say it is definitely worthwhile if you're trying to save space and reduce compile times.

On interest I can submit a PR. Otherwise, it was real fun writing C++ again after a long while.

cc @dmah42

[dmah42]

I have no strong opinion here. I'm happy to take a PR as long as nano bind is broadly available.

[nicholasjng]

It's designed to work with all major OSs and compilers, along with Python >=3.8. It might require the move to C++17 for the wheel target, but presumably not much else, if anything. (I only compiled the nanobind cc_library itself with -std=c++17 and not the whole wheel, so I am unsure if we could possibly become even faster compiling benchmark for C++17, too.)

There are some features, namely nb::bind_map and export_values return values, that have not made it into a release yet, but once there is a new release, the headers and the library can even be grabbed pre-compiled from PyPI via pip install nanobind - then, nanobind could even be used for a cmake-only build e.g. using scikit-build instead of Bazel + setuptools.

I'll overhaul and submit for review then. Thanks!

[nicholasjng]

Closed by #1526.