Cythonize call loop

[goodboy]

Consider this a working initial draft to begin our foray into cython-land.
The benchmark suite shows a x2 speedup with this cythonized version of _multicall 👍

[goodboy]

Apparently setup_requires can be used according to this SO answer if we want to always rely on cython as a dep. Looking for opinions here.

[RonnyPfannschmidt]

please dont commit the c sources to the git repo, also not exactly what i had in mind for cythonizing :(

[goodboy]

please dont commit the c sources to the git repo

Done.

also not exactly what i had in mind for cythonizing :(

We can always make it better 👍.
I really just wanted to do this to see how much of a speedup we'd get.

[RonnyPfannschmidt]

the biggest problem i see is we broke sane pypy support - c extensions are just not the way to go)

[RonnyPfannschmidt]

it looks like this invocation got accidentally plucked incorrectly

[goodboy]

I actually did this on purpose so I could pass additional args without having to pass testing/benchmark every time. Should I put it back?

[goodboy]

the biggest problem i see is we broke sane pypy support - c extensions are just not the way to go)

So what did you mean by cythonize then in our previous discussions?

Also, with regard to pypy I agree it seems cython doesn't have great support although it seems the pypy build passsed?

We can keep the original pure python implementation as well and then only import the cythonized module when running on cpython?

[RonnyPfannschmidt]

i recall that cython had a mode where the code was pure python * a separate file with annotation

[nicoddemus]

I'm not sure the speed up is worth it; I have the feeling that the bottleneck in applications using pluggy is in the actual implementation of the hooks, not the time spent in the multi-call mechanism.

[RonnyPfannschmidt]

@nicoddemus the speed of the fixture mechanism and the hook calling mechanism is actually critical in some bits

the switch from kwargs to positional args already did a lot to safe time, and i suspect there is more to gain

i recall one of the reasons why pytest doesn't integrate xunit with fixtures is, that the fixtures where too expensive

[nicoddemus]

i recall one of the reasons why pytest doesn't integrate xunit with fixtures is, that the fixtures where too expensive

Probably because the initial implementation always added a setup/teardown fixture for all levels (function/class/module) regardless if there were an implementation of setUp/tearDown in the unittest class; this could be circumvented by only creating the autouse-fixtures when there is a respective xUnit method.

But I digress, if indeed this is a bottleneck let's see if we can squeeze some more performance from it. 👍

[goodboy]

i recall that cython had a mode where the code was pure python

Yes there is pure python mode though speed gains are much more negligible.
I'm happy to try this out - including the approach using the external .pxd file.

[hpk42]

Is there any project where calling hooks is shown to be a speed issue?
I am using pluggy in 5 different projects and there is not the slightest problem.
Last time i profiled with pytest the multicall-machinery did not show up in any significant way.
So i suggest to close this issue and not bother for optimizing unless there is clear evidence it's needed (and even then, there might be easy wins without going full cython).

[goodboy]

@hpk42 I totally agree. Forgive my premature cythonization ;)