Reusing the args tuple for PyCFunction calls?

[Fidget-Spinner]

I noticed this call doesn't specialize because json.loads is calls with METH_VARARGS (aka PyCFunction) and PyCFunctionWithKeywords don't specialize.

Specializing for that isn't going to give us a large speedup (if any at all) because the temporary args tuple still needs to be constructed at every call. The args tuple is then usually deallocated after the call.

What if:

At compile time, we find the call with the most number of arguments in the code object, then pre-allocate an empty tuple of that size and attach it to the code object.

At runtime, we re-use this preallocated tuple object on every call, avoiding most tuple allocations (maybe?).

Has anyone tried this before and found that it speeds up anything? In theory, this should make non-vectorcall calls nearly as fast as vectorcall.

Note:
We should not apply this to calls to python functions because it's already done with how inlined Python call frames currently work in the CALL instruction.

Edit:
I removed my rambling about json.load because it was inaccurate. See this comment.

[da-woods]

What happens if json.loads consists of

Py_INCREF(args);
return args;

?

[Fidget-Spinner]

I don't think that code is very common, but there will be a guard to check that Py_REFCNT(args) == 1. Else Python will create a new args tuple.

[Fidget-Spinner]

Your comment made me think deeper, and I realised that the specialiser could be useful here. Instead of one tuple per code object, we could dynamically create a tuple at every specialisation site only for such C functions. Then if we detect that it exhibits behavior like you suggested (ie Py_REFCNT(args) != 1, we just de-specialize back to the original CALL instruction.

[Fidget-Spinner]

Sorry, I just noticed that json.loads was a bad example. I forgot that it's a Python function, so the CALL instruction which inlines Python stack frames just needs to copy over args from the stack into locals of the new frame. There's no temporary tuple created.

A better example would be calls to any C function/type that still doesn't support vectorcall (PyCFunction and PyCFunctionWithKeywords).

[markshannon]

Ultimately we want to use a more efficient interface, #374.

In the short to medium term, the cache of tuples could work, but where would it live?
We don't want to store strong references in the inline caches, so there would need to be a strong reference to it somewhere.

[Fidget-Spinner]

Ultimately we want to use a more efficient interface, #374.

Agreed. I think there's some value in this because this comes free of charge without any changes in code required by other people.

I was originally thinking of storing strong references in the inline cache, but the cleanup would be rather complicated.

I'm now considering storing the compile-time determined tuple as a per-code object entry.

At specialization time, we create two PyCFunction specialisations, CALL_BUILTIN_PYCFUNCTION and CALL_BUILTIN_PYCFUNCTION_KWDS. These specializations will re-use the args tuple.

They should also de-opt if they see that Py_REFCNT(args) != 1. But this should be done only after the call. This will prevent things similar to this scenario, while allowing functions that play nice to reap the benefits.

The only pain point is that de-opt will need custom handling since it's not side-effect free. I'm thinking of manually handling it instead of letting the current code do it.

[kumaraditya303]

Another thing is that these tuples have strong reference so there are a lot of incref -> call -> decref whereas vectorcall uses borrowed references so it avoids most refcounting. I think the same might work for tuples if they hold borrowed references and a special decref can be used which checks if something has a ref to tuple to incref all its items else just push the tuple to freelist with memset 0.

[Fidget-Spinner]

Yup. Also since tuples currently already have freelist for size < 20. I'll need to try something smarter to see any sort of speedup. Your borrowed references idea is probably the way to go. I don't think reusing tuples here will beat the current free list. However, we can probably delay the args incref to only when we detect the tuple is being used for other reasons.

The more I think about this idea, the more pessimistic I am on the speedup. However, I'll write the code and benchmark it to see for myself.

[gvanrossum]

I guess the refcnt check will handle recursive calls and calls from multiple threads at once? I'm not sure how this would interact with nogil but we'll have to cross that bridge when we get to it.