Skip to content

bpo-45367: Specialize BINARY_MULTIPLY #28727

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Oct 14, 2021
Merged

bpo-45367: Specialize BINARY_MULTIPLY #28727

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
3c85756
Implement specialized opcodes for BINARY_MULTIPLY
sweeneyde Oct 5, 2021
300a0ca
📜🤖 Added by blurb_it.
blurb-it[bot] Oct 5, 2021
d266912
Resolve merge conflicts
sweeneyde Oct 5, 2021
c64540a
Apply changes from code review
sweeneyde Oct 5, 2021
5a44ecc
Allow float/int mixtures is BINARY_MULTIPLY_FLOAT
sweeneyde Oct 5, 2021
aea424e
Minor cleanups and comments
sweeneyde Oct 5, 2021
37d3716
Revert the float/int mixtures
sweeneyde Oct 7, 2021
903ff9c
typo: sum --> prod
sweeneyde Oct 7, 2021
7aecc97
Merge branch 'main' into multiply
sweeneyde Oct 10, 2021
31c3bb9
Merge main into multiply
sweeneyde Oct 13, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion Include/internal/pycore_code.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,7 +307,8 @@ int _Py_Specialize_StoreAttr(PyObject *owner, _Py_CODEUNIT *instr, PyObject *nam
int _Py_Specialize_LoadGlobal(PyObject *globals, PyObject *builtins, _Py_CODEUNIT *instr, PyObject *name, SpecializedCacheEntry *cache);
int _Py_Specialize_LoadMethod(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name, SpecializedCacheEntry *cache);
int _Py_Specialize_BinarySubscr(PyObject *sub, PyObject *container, _Py_CODEUNIT *instr);
int _Py_Specialize_BinaryAdd(PyObject *sub, PyObject *container, _Py_CODEUNIT *instr);
int _Py_Specialize_BinaryAdd(PyObject *left, PyObject *right, _Py_CODEUNIT *instr);
int _Py_Specialize_BinaryMultiply(PyObject *left, PyObject *right, _Py_CODEUNIT *instr);

#define PRINT_SPECIALIZATION_STATS 0
#define PRINT_SPECIALIZATION_STATS_DETAILED 0
Expand Down
1 change: 1 addition & 0 deletions Include/internal/pycore_long.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ static inline PyObject* _PyLong_GetOne(void)
{ return __PyLong_GetSmallInt_internal(1); }

PyObject *_PyLong_Add(PyLongObject *left, PyLongObject *right);
PyObject *_PyLong_Multiply(PyLongObject *left, PyLongObject *right);

#ifdef __cplusplus
}
Expand Down
57 changes: 30 additions & 27 deletions Include/opcode.h
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Lib/opcode.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,9 @@ def jabs_op(name, op):
"BINARY_ADD_FLOAT",
"BINARY_ADD_UNICODE",
"BINARY_ADD_UNICODE_INPLACE_FAST",
"BINARY_MULTIPLY_ADAPTIVE",
"BINARY_MULTIPLY_INT",
"BINARY_MULTIPLY_FLOAT",
"BINARY_SUBSCR_ADAPTIVE",
"BINARY_SUBSCR_LIST_INT",
"BINARY_SUBSCR_TUPLE_INT",
Expand Down
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Core and Builtins/2021-10-05-03-49-07.bpo-45367._astoU.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Specialized the ``BINARY_MULTIPLY`` opcode to ``BINARY_MULTIPLY_INT`` and ``BINARY_MULTIPLY_FLOAT`` using the PEP 659 machinery.
13 changes: 9 additions & 4 deletions Objects/longobject.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3593,13 +3593,11 @@ k_lopsided_mul(PyLongObject *a, PyLongObject *b)
return NULL;
}

static PyObject *
long_mul(PyLongObject *a, PyLongObject *b)
PyObject *
_PyLong_Multiply(PyLongObject *a, PyLongObject *b)
{
PyLongObject *z;

CHECK_BINOP(a, b);

/* fast path for single-digit multiplication */
if (IS_MEDIUM_VALUE(a) && IS_MEDIUM_VALUE(b)) {
stwodigits v = medium_value(a) * medium_value(b);
Expand All @@ -3616,6 +3614,13 @@ long_mul(PyLongObject *a, PyLongObject *b)
return (PyObject *)z;
}

static PyObject *
long_mul(PyLongObject *a, PyLongObject *b)
{
CHECK_BINOP(a, b);
return _PyLong_Multiply(a, b);
}

/* Fast modulo division for single-digit longs. */
static PyObject *
fast_mod(PyLongObject *a, PyLongObject *b)
Expand Down
62 changes: 61 additions & 1 deletion Python/ceval.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1933,14 +1933,73 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr
}

TARGET(BINARY_MULTIPLY) {
PREDICTED(BINARY_MULTIPLY);
STAT_INC(BINARY_MULTIPLY, unquickened);
PyObject *right = POP();
PyObject *left = TOP();
PyObject *res = PyNumber_Multiply(left, right);
Py_DECREF(left);
Py_DECREF(right);
SET_TOP(res);
if (res == NULL)
if (res == NULL) {
goto error;
}
DISPATCH();
}

TARGET(BINARY_MULTIPLY_ADAPTIVE) {
if (oparg == 0) {
PyObject *left = SECOND();
PyObject *right = TOP();
next_instr--;
if (_Py_Specialize_BinaryMultiply(left, right, next_instr) < 0) {
goto error;
}
DISPATCH();
}
else {
STAT_INC(BINARY_MULTIPLY, deferred);
UPDATE_PREV_INSTR_OPARG(next_instr, oparg - 1);
STAT_DEC(BINARY_MULTIPLY, unquickened);
JUMP_TO_INSTRUCTION(BINARY_MULTIPLY);
}
}

TARGET(BINARY_MULTIPLY_INT) {
PyObject *left = SECOND();
PyObject *right = TOP();
DEOPT_IF(!PyLong_CheckExact(left), BINARY_MULTIPLY);
DEOPT_IF(!PyLong_CheckExact(right), BINARY_MULTIPLY);
STAT_INC(BINARY_MULTIPLY, hit);
record_hit_inline(next_instr, oparg);
PyObject *prod = _PyLong_Multiply((PyLongObject *)left, (PyLongObject *)right);
SET_SECOND(prod);
Py_DECREF(right);
Py_DECREF(left);
STACK_SHRINK(1);
if (prod == NULL) {
goto error;
}
DISPATCH();
}

TARGET(BINARY_MULTIPLY_FLOAT) {
PyObject *left = SECOND();
PyObject *right = TOP();
DEOPT_IF(!PyFloat_CheckExact(left), BINARY_MULTIPLY);
DEOPT_IF(!PyFloat_CheckExact(right), BINARY_MULTIPLY);
STAT_INC(BINARY_MULTIPLY, hit);
record_hit_inline(next_instr, oparg);
double dprod = ((PyFloatObject *)left)->ob_fval *
((PyFloatObject *)right)->ob_fval;
PyObject *prod = PyFloat_FromDouble(dprod);
SET_SECOND(prod);
Py_DECREF(right);
Py_DECREF(left);
STACK_SHRINK(1);
if (prod == NULL) {
goto error;
}
DISPATCH();
}

Expand Down Expand Up @@ -4954,6 +5013,7 @@ MISS_WITH_CACHE(LOAD_GLOBAL)
MISS_WITH_CACHE(LOAD_METHOD)
MISS_WITH_OPARG_COUNTER(BINARY_SUBSCR)
MISS_WITH_OPARG_COUNTER(BINARY_ADD)
MISS_WITH_OPARG_COUNTER(BINARY_MULTIPLY)

binary_subscr_dict_error:
{
Expand Down
44 changes: 22 additions & 22 deletions Python/opcode_targets.h
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

33 changes: 33 additions & 0 deletions Python/specialize.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ _Py_GetSpecializationStats(void) {
err += add_stat_dict(stats, LOAD_GLOBAL, "load_global");
err += add_stat_dict(stats, LOAD_METHOD, "load_method");
err += add_stat_dict(stats, BINARY_ADD, "binary_add");
err += add_stat_dict(stats, BINARY_MULTIPLY, "binary_multiply");
err += add_stat_dict(stats, BINARY_SUBSCR, "binary_subscr");
err += add_stat_dict(stats, STORE_ATTR, "store_attr");
if (err < 0) {
Expand Down Expand Up @@ -180,6 +181,7 @@ _Py_PrintSpecializationStats(void)
print_stats(out, &_specialization_stats[LOAD_GLOBAL], "load_global");
print_stats(out, &_specialization_stats[LOAD_METHOD], "load_method");
print_stats(out, &_specialization_stats[BINARY_ADD], "binary_add");
print_stats(out, &_specialization_stats[BINARY_MULTIPLY], "binary_multiply");
print_stats(out, &_specialization_stats[BINARY_SUBSCR], "binary_subscr");
print_stats(out, &_specialization_stats[STORE_ATTR], "store_attr");
if (out != stderr) {
Expand Down Expand Up @@ -230,6 +232,7 @@ static uint8_t adaptive_opcodes[256] = {
[LOAD_GLOBAL] = LOAD_GLOBAL_ADAPTIVE,
[LOAD_METHOD] = LOAD_METHOD_ADAPTIVE,
[BINARY_ADD] = BINARY_ADD_ADAPTIVE,
[BINARY_MULTIPLY] = BINARY_MULTIPLY_ADAPTIVE,
[BINARY_SUBSCR] = BINARY_SUBSCR_ADAPTIVE,
[STORE_ATTR] = STORE_ATTR_ADAPTIVE,
};
Expand All @@ -240,6 +243,7 @@ static uint8_t cache_requirements[256] = {
[LOAD_GLOBAL] = 2, /* _PyAdaptiveEntry and _PyLoadGlobalCache */
[LOAD_METHOD] = 3, /* _PyAdaptiveEntry, _PyAttrCache and _PyObjectCache */
[BINARY_ADD] = 0,
[BINARY_MULTIPLY] = 0,
[BINARY_SUBSCR] = 0,
[STORE_ATTR] = 2, /* _PyAdaptiveEntry and _PyAttrCache */
};
Expand Down Expand Up @@ -1188,3 +1192,32 @@ _Py_Specialize_BinaryAdd(PyObject *left, PyObject *right, _Py_CODEUNIT *instr)
assert(!PyErr_Occurred());
return 0;
}

int
_Py_Specialize_BinaryMultiply(PyObject *left, PyObject *right, _Py_CODEUNIT *instr)
{
if (!Py_IS_TYPE(left, Py_TYPE(right))) {
SPECIALIZATION_FAIL(BINARY_MULTIPLY, SPEC_FAIL_DIFFERENT_TYPES);
goto fail;
}
if (PyLong_CheckExact(left)) {
*instr = _Py_MAKECODEUNIT(BINARY_MULTIPLY_INT, saturating_start());
goto success;
}
else if (PyFloat_CheckExact(left)) {
*instr = _Py_MAKECODEUNIT(BINARY_MULTIPLY_FLOAT, saturating_start());
goto success;
}
else {
SPECIALIZATION_FAIL(BINARY_MULTIPLY, SPEC_FAIL_OTHER);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this should be refined further. In another PR, though.

Copy link
Member

@Fidget-Spinner Fidget-Spinner Oct 5, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. I don't think it will touch pyperformance, but it seems that in our own test suite almost 10% aren't hits #28727 (comment).

This probably varies wildly, I'd imagine test_string has lots of str * num too.
Oh gosh I just realized I'd read the entire PR as BINARY_ADD not BINARY_MULTIPLY. Please ignore my delusional ramblings.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of the non-hits are deferred, not specialization failure, so I think that's because the nature of a lot of test code is to only be run once, without many tight loops.

Copy link
Member

@Fidget-Spinner Fidget-Spinner Oct 5, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out Dennis. I can't believe I missed that :). That means nearly 100% actual specialization on hot code. Hooray!

Off-topic: I've recently wondered if pyperformance is somewhat out of touch (no offence intended to its contributors). Many of its benchmarks (nbody, nqueens, etc.) have been found by the JS folks to not be realistic. The Pyston benchmark suite seems way more comprehensive.

Copy link
Member

@markshannon markshannon Oct 5, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"deferred" means that the ADAPTIVE instruction is counting down until its next specialization attempt.
This only happens after a specialization failure.

Look at the numbers. deferred ≈ sum(specialization-failures) * ADAPTIVE_CACHE_BACKOFF

}
fail:
STAT_INC(BINARY_MULTIPLY, specialization_failure);
assert(!PyErr_Occurred());
*instr = _Py_MAKECODEUNIT(_Py_OPCODE(*instr), ADAPTIVE_CACHE_BACKOFF);
return 0;
success:
STAT_INC(BINARY_MULTIPLY, specialization_success);
assert(!PyErr_Occurred());
return 0;
}