Add benchmarks for C function call overhead of various alternatives

[mdboom]

It might be useful to benchmark the overhead of making C extension function calls using some of the popular approaches.

There is already a set of benchmarks available, with details in this blog post. It checks the function call overhead of the following:

• extension: Python/C API with PyArg_ParseTuple
• extension_manunpack: Python/C API with PyArg_UnpackTuple
• cython
• swig
• boost
• ctypes

I hacked around with this during the PyCon sprints, and made my own fork of these benchmarks that:

• Adds error checking to extension_manunpack (seems like an unfair advantage without it)
• Adds a benchmark for the METH_FASTCALL API.
• Uses the current Cython git main to have something that works with py3.11
• Uses more iterations and all of pyperf's advice for stable benchmarking.

With that, there are some interesting results:

1. The benchmarks just call a C extension function 5,000,000 time in a Python loop, so for many of these, the speedup is most likely just to improved loop performance. Maybe we want to control for that.
2. SWIG adds a Python wrapper around everything, so it's not surprising it sees the most significant speedup in 3.11
3. There seems to be a regression in ctypes, which is maybe concerning...?

[FreddieWitherden]

Interesting results! One thing to note on the ctypes side is that, at least in my experience, explicitly setting the argument/return types is associated with a very real degree of overhead in lieu of letting ctypes deduce everything on the fly. It might be interesting to include this in the benchmark also.

[mdboom]

Great callout, @FreddieWitherden. In this case, this is letting ctypes deduce everything on the fly, but I agree, the manual typing approach should be benchmarked as well.

[mdboom]

With the ctypes performance regression in python/cpython#92356 fixed, and a new benchmark added for "ctypes with specified argument types" the results are now:

[FreddieWitherden]

Great to see things with ctypes are now faster with 3.11 than 3.10!

In terms of the argument spec, my understanding is that the overhead comes from the fact we end up calling one function (usually a Python function) per argument to perform the coercion. Overhead therefore scales with ~ nargs. Given many arguments are 'simple' (integers, floats, pointers to structures etc.) it should be possible to inline and special-case most of the standard types (https://github.com/python/cpython/blob/main/Modules/_ctypes/callproc.c#L1191 is the source of the overhead).

[wjakob]

Nanobind also contains a few benchmarks of this type: https://github.com/wjakob/nanobind (It uses vectorcalls internally, which are unfortunately not part of the stable API)

[gvanrossum]

In terms of the argument spec, my understanding is that the overhead comes from the fact we end up calling one function (usually a Python function) per argument to perform the coercion. Overhead therefore scales with ~ nargs.

Can you create a new issue to discuss speeding up ctypes?

[Fidget-Spinner]

@mdboom mixing some different argument/call forms into the bench might be interesting too. Looking at your benchmarks, they're all fixed argument counts. IIRC, the 3.11 specializer doesn't optimize for a few call types. So C function calls with *args/**kwargs function may have suffered slightly. How common these call signatures are, I have no clue.

[markshannon]

@wjakob Vectorcall is explicitly designed to be usable by third-part extensions, so can be considered part of the stable API (and ABI), https://peps.python.org/pep-0590