bpo-44725 : expose specialization stats in python

[iritkatriel]

This PR exposes the specialisation stats as a python dictionary. It makes them easier to work with (including delta for a code snippet and unit tests for the specialisation).

I suggest that we also:

1. Remove the print function.
2. Define SPECIALIZATION_STATS and SPECIALIZATION_STATS_DETAILED to 1 if Py_DEBUG
3. Actually, remove SPECIALIZATION_STATS_DETAILED and make the detailed stats either always be in the output, or controlled by a parameter to get_specialization_stats.

Let me know if that sounds reasonable.

https://bugs.python.org/issue44725

>>> import _opcode, print
>>> pprint.pprint(_opcode.get_specialization_stats())
{'binary_subscr': {'deferred': 1545,
                   'deopt': 4,
                   'detailed': {(<class 'mappingproxy'>, 'asend', 'not list|tuple|dict'): 1,
                                (<class 'str'>, -3, 'not list|tuple|dict'): 1,
                                (<class 'str'>, -1, 'not list|tuple|dict'): 2,
                                (<class 'str'>, 0, 'not list|tuple|dict'): 8,
                                (<class 'str'>, 2, 'not list|tuple|dict'): 1,
                                (<class 'enum._EnumDict'>, '_ignore_', 'not list|tuple|dict'): 1,
                                (<class 'sre_parse.SubPattern'>, 0, 'not list|tuple|dict'): 8,
                                (<class 'str'>, 7, 'not list|tuple|dict'): 1,
                                (<class 'str'>, 19, 'not list|tuple|dict'): 1,
                                (<class 'str'>, 26, 'not list|tuple|dict'): 1,
                                (<class 'mappingproxy'>, 'from_bytes', 'not list|tuple|dict'): 1},
                   'hit': 870,
                   'miss': 12,
                   'specialization_failure': 49,
                   'specialization_success': 49,
                   'unquickened': 1879},
 'load_attr': {'deferred': 3206,
               'deopt': 24,
               'detailed': {(<class '_frozen_importlib.ModuleSpec'>, 'cached', 'property'): 2,
                            (<class '_frozen_importlib.ModuleSpec'>, 'has_location', 'property'): 2,
                            (<class '_frozen_importlib.ModuleSpec'>, 'parent', 'property'): 2,
                            (<class 'function'>, '__dict__', 'overriding descriptor'): 4,
                            (<class 'sys.flags'>, 'optimize', 'non-object slot'): 2,
                            (<class 'type'>, '_ORIGIN', '__getattribute__ overridden'): 1,
                            (<class 'type'>, 'find_spec', '__getattribute__ overridden'): 3,
                            (<class 'sys.flags'>, 'verbose', 'non-object slot'): 10,
                            (<class 'os.stat_result'>, 'st_mode', 'non-object slot'): 2,
                            (<class 'os.stat_result'>, 'st_mtime', 'non-object slot'): 3,
                            (<class 'os.stat_result'>, 'st_size', 'non-object slot'): 1,
                            (<class 'sys.flags'>, 'ignore_environment', 'non-object slot'): 2,
                            (<class 'module'>, '__dict__', 'index out of range'): 1,
                            (<class 'type'>, '__dict__', '__getattribute__ overridden'): 8,
                            (<class 'super'>, '__new__', '__getattribute__ overridden'): 3,
                            (<class 'type'>, '__mro__', '__getattribute__ overridden'): 3,
                            (<class 'type'>, '__new__', '__getattribute__ overridden'): 5,
                            (<class 'classmethod'>, '__func__', 'non-object slot'): 2,
                            (<class 'function'>, '__name__', 'overriding descriptor'): 2,
                            (<class 'frame'>, 'f_globals', 'overriding descriptor'): 1,
                            (<class 'enum.EnumType'>, 'VAR_KEYWORD', '__getattribute__ overridden'): 1,
                            (<class 'enum.EnumType'>, 'VAR_POSITIONAL', '__getattribute__ overridden'): 1,
                            (<class 'enum.EnumType'>, '__new__', '__getattribute__ overridden'): 4,
                            (<class 'enum.EnumType'>, '_member_map_', '__getattribute__ overridden'): 7,
                            (<class 'enum.EnumType'>, '_member_names_', '__getattribute__ overridden'): 4,
                            (<class 'enum.auto'>, 'value', 'no dict or not a dict'): 2,
                            (<class 'enum.EnumType'>, '_member_type_', '__getattribute__ overridden'): 7,
                            (<class 'NoneType'>, '__doc__', 'non descriptor'): 1,
                            (<class 'NoneType'>, '__new__', 'non descriptor'): 1,
                            (<class 'enum.EnumType'>, '__dict__', '__getattribute__ overridden'): 1,
                            (<class 'enum.EnumType'>, '__mro__', '__getattribute__ overridden'): 1,
                            (<class 'enum.EnumType'>, '__name__', '__getattribute__ overridden'): 1,
                            (<class 'enum.EnumType'>, '_generate_next_value_', '__getattribute__ overridden'): 1,
                            (<class 'enum.EnumType'>, '_new_member_', '__getattribute__ overridden'): 1,
                            (<class 'enum.EnumType'>, '_use_args_', '__getattribute__ overridden'): 2,
                            (<class 'enum.EnumType'>, '_value2member_map_', '__getattribute__ overridden'): 1,
                            (<enum 'FlagBoundary'>, '__init__', 'method'): 1,
                            (<class 'type'>, '__name__', '__getattribute__ overridden'): 1,
                            (<flag 'RegexFlag'>, '_value_', 'negative offset'): 1,
                            (<class 'enum.EnumType'>, '_iter_member_by_def_', '__getattribute__ overridden'): 1,
                            (<class 'sre_constants._NamedIntConstant'>, 'name', 'negative offset'): 2,
                            (<class 'list'>, 'append', 'method'): 10,
                            (<class 'sre_parse.SubPattern'>, 'append', 'method'): 1,
                            (<class 'sre_parse.Tokenizer'>, 'get', 'method'): 1,
                            (<class 'sre_parse.Tokenizer'>, 'match', 'method'): 2,
                            (<class 'type'>, '__call__', '__getattribute__ overridden'): 1,
                            (<class 'type'>, '__repr__', '__getattribute__ overridden'): 11,
                            (<class 'type'>, '_fields', '__getattribute__ overridden'): 1,
                            (<class 'builtin_function_or_method'>, '__call__', 'method'): 1,
                            (<class 'enum.EnumType'>, 'KEYWORD_ONLY', '__getattribute__ overridden'): 1,
                            (<class 'enum.EnumType'>, 'POSITIONAL_ONLY', '__getattribute__ overridden'): 1,
                            (<class 'enum.EnumType'>, 'POSITIONAL_OR_KEYWORD', '__getattribute__ overridden'): 1,
                            (<class 'pprint.PrettyPrinter'>, '_dispatch', 'attribute not in dict'): 1,
                            (<class '_io.TextIOWrapper'>, 'write', 'method'): 1,
                            (<class 'type'>, 'copy', '__getattribute__ overridden'): 4},
               'hit': 5628,
               'miss': 178,
               'specialization_failure': 120,
               'specialization_success': 157,
               'unquickened': 4886},
 'load_global': {'deferred': 3370,
                 'deopt': 199,
                 'hit': 7880,
                 'miss': 1538,
                 'specialization_failure': 0,
                 'specialization_success': 471,
                 'unquickened': 11157}}

https://bugs.python.org/issue44725

[Fidget-Spinner]

This PR exposes the specialisation stats as a python dictionary

Awesome! I was just wondering if there was some better way to collect data and this PR does that :). Thanks!

[markshannon]

I don't object to exposing the stats, but be warned that accessing them via Python is going to distort the results.
For accurate stats, they should be dumped to a file (or command line) on program exit.

[iritkatriel]

I don't object to exposing the stats, but be warned that accessing them via Python is going to distort the results.
For accurate stats, they should be dumped to a file (or command line) on program exit.

Right, I'm thinking about calculating deltas that exclude any of the processing. So the call to fetch the numbers can contaminate the results (though arguably less than startup and shutdown of the whole program).

We can leave the printing at the end as well.

[markshannon]

Can we derive this list from elsewhere?
Maybe move this list to opcode.py and generate it much like we generate the specialized instructions and jump tables?

[iritkatriel]

I agree with this suggestion and have started working on it. It's a fairly large change to the existing code, so might be appropriate to do in a separate PR.

[iritkatriel]

There is a slight complication with this due to NEED_OPCODE_JUMP_TABLES. Is that temporary?

The issue is this:
compile.c does

#define NEED_OPCODE_JUMP_TABLES
#include "opcode.h"               // EXTENDED_ARG

so if I want specialise.c to include opcode.h I need to put the define there as well. But that is beginning to be messy. So maybe we move the define into opcode.h so that it can be included more than once?

[iritkatriel]

Another option is that I add a specialize.h file with specialization related definitions, and a Tools/scripts/generate_specialize.h that creates it.

[markshannon]

Why "detailed"? This is named "fails" in print_stats().

[markshannon]

LGTM

[markshannon]

Thanks @iritkatriel