C-API for signalling monitoring events

[scoder]

Feature or enhancement

Proposal:

Language implementations for the CPython runtime (Cython, JIT compilers, regular expression engines, template engines, etc.) need a way to signal PEP-669 monitoring events to the registered listeners.

1. We need a way to create events and inject them into the monitoring system. Since events have more than one signature, we might end up needing more than one C-API function for this.

2. We need a way to map 3D source code positions (file name, line, character) to 1D integer offsets. Code objects help, but branches might cross source file boundaries, so that's more tricky. For many use cases, a mapping between (line, character) positions and an integer offset would probably suffice, although templating languages usually provide some kind of include commands, as does Cython. There should be some help in the C-API for building up such a mapping.

The reason why I think we need CPython's help for mapping indices is that both sides, listeners and even producers, need to agree on the same mapping. Sadly, the events don't include line/character positions directly but only a single integer. So event producers need a way to produce a number that event listeners like coverage analysers can map back to a source code position.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/pep-669-low-impact-monitoring-for-cpython/13018/61?u=scoder

Linked PRs

• gh-111997: C-API for signalling monitoring events #116413
• gh-111997: Fix argument count for LINE event and clarify type of argument counts. #119179
• [3.13] gh-111997: Fix argument count for LINE event and clarify type of argument counts. (GH-119179) #119575
• gh-111997: update what's new in 3.13 re C API for monitoring events #123822
• [3.13] gh-111997: update what's new in 3.13 re C API for monitoring events (GH-123822) #123997

[scoder]

@encukou @markshannon @vstinner

[vstinner]

Do you want to propose an API for that?

[drdavella]

PEP-669 seems to discuss a mechanism to modify the bytestream for the purpose of implementing monitoring. Would access to this mechanism also be provided as part of this API? I haven't seen too much mention of this feature elsewhere.

Source: https://peps.python.org/pep-0669/#rationale

The quickening mechanism provided by PEP 659 provides a way to dynamically modify executing Python bytecode. These modifications have little cost beyond the parts of the code that are modified and a relatively low cost to those parts that are modified. We can leverage this to provide an efficient mechanism for monitoring that was not possible in 3.10 or earlier.

IMO it would be very useful for an API of this nature to expose low-level interpreter state to monitoring tools. For example, if we are able to see events like INSTRUCTION, it would also be useful to have access to the operands involved.

[scoder]

Do you want to propose an API for that?

In order to map source/line/column information to an offset, we need a way to build a co_linetable for a code object: https://github.com/python/cpython/blob/main/Objects/locations.md
That's probably the most complex thing to do here. IIUC, CPython implements this in C here:

cpython/Python/assemble.c

Lines 190 to 444 in d67f947

	/* Code location emitting code. See locations.md for a description of the format. */
	#define MSB 0x80
	static void
	write_location_byte(struct assembler* a, int val)
	{
	PyBytes_AS_STRING(a->a_linetable)[a->a_location_off] = val&255;
	a->a_location_off++;
	}
	static uint8_t *
	location_pointer(struct assembler* a)
	{
	return (uint8_t *)PyBytes_AS_STRING(a->a_linetable) +
	a->a_location_off;
	}
	static void
	write_location_first_byte(struct assembler* a, int code, int length)
	{
	a->a_location_off += write_location_entry_start(
	location_pointer(a), code, length);
	}
	static void
	write_location_varint(struct assembler* a, unsigned int val)
	{
	uint8_t *ptr = location_pointer(a);
	a->a_location_off += write_varint(ptr, val);
	}
	static void
	write_location_signed_varint(struct assembler* a, int val)
	{
	uint8_t *ptr = location_pointer(a);
	a->a_location_off += write_signed_varint(ptr, val);
	}
	static void
	write_location_info_short_form(struct assembler* a, int length, int column, int end_column)
	{
	assert(length > 0 && length <= 8);
	int column_low_bits = column & 7;
	int column_group = column >> 3;
	assert(column < 80);
	assert(end_column >= column);
	assert(end_column - column < 16);
	write_location_first_byte(a, PY_CODE_LOCATION_INFO_SHORT0 + column_group, length);
	write_location_byte(a, (column_low_bits << 4) | (end_column - column));
	}
	static void
	write_location_info_oneline_form(struct assembler* a, int length, int line_delta, int column, int end_column)
	{
	assert(length > 0 && length <= 8);
	assert(line_delta >= 0 && line_delta < 3);
	assert(column < 128);
	assert(end_column < 128);
	write_location_first_byte(a, PY_CODE_LOCATION_INFO_ONE_LINE0 + line_delta, length);
	write_location_byte(a, column);
	write_location_byte(a, end_column);
	}
	static void
	write_location_info_long_form(struct assembler* a, location loc, int length)
	{
	assert(length > 0 && length <= 8);
	write_location_first_byte(a, PY_CODE_LOCATION_INFO_LONG, length);
	write_location_signed_varint(a, loc.lineno - a->a_lineno);
	assert(loc.end_lineno >= loc.lineno);
	write_location_varint(a, loc.end_lineno - loc.lineno);
	write_location_varint(a, loc.col_offset + 1);
	write_location_varint(a, loc.end_col_offset + 1);
	}
	static void
	write_location_info_none(struct assembler* a, int length)
	{
	write_location_first_byte(a, PY_CODE_LOCATION_INFO_NONE, length);
	}
	static void
	write_location_info_no_column(struct assembler* a, int length, int line_delta)
	{
	write_location_first_byte(a, PY_CODE_LOCATION_INFO_NO_COLUMNS, length);
	write_location_signed_varint(a, line_delta);
	}
	#define THEORETICAL_MAX_ENTRY_SIZE 25 /* 1 + 6 + 6 + 6 + 6 */
	static int
	write_location_info_entry(struct assembler* a, location loc, int isize)
	{
	Py_ssize_t len = PyBytes_GET_SIZE(a->a_linetable);
	if (a->a_location_off + THEORETICAL_MAX_ENTRY_SIZE >= len) {
	assert(len > THEORETICAL_MAX_ENTRY_SIZE);
	RETURN_IF_ERROR(_PyBytes_Resize(&a->a_linetable, len*2));
	}
	if (loc.lineno < 0) {
	write_location_info_none(a, isize);
	return SUCCESS;
	}
	int line_delta = loc.lineno - a->a_lineno;
	int column = loc.col_offset;
	int end_column = loc.end_col_offset;
	assert(column >= -1);
	assert(end_column >= -1);
	if (column < 0 || end_column < 0) {
	if (loc.end_lineno == loc.lineno || loc.end_lineno == -1) {
	write_location_info_no_column(a, isize, line_delta);
	a->a_lineno = loc.lineno;
	return SUCCESS;
	}
	}
	else if (loc.end_lineno == loc.lineno) {
	if (line_delta == 0 && column < 80 && end_column - column < 16 && end_column >= column) {
	write_location_info_short_form(a, isize, column, end_column);
	return SUCCESS;
	}
	if (line_delta >= 0 && line_delta < 3 && column < 128 && end_column < 128) {
	write_location_info_oneline_form(a, isize, line_delta, column, end_column);
	a->a_lineno = loc.lineno;
	return SUCCESS;
	}
	}
	write_location_info_long_form(a, loc, isize);
	a->a_lineno = loc.lineno;
	return SUCCESS;
	}
	static int
	assemble_emit_location(struct assembler* a, location loc, int isize)
	{
	if (isize == 0) {
	return SUCCESS;
	}
	while (isize > 8) {
	RETURN_IF_ERROR(write_location_info_entry(a, loc, 8));
	isize -= 8;
	}
	return write_location_info_entry(a, loc, isize);
	}
	static int
	assemble_location_info(struct assembler *a, instr_sequence *instrs,
	int firstlineno)
	{
	a->a_lineno = firstlineno;
	location loc = NO_LOCATION;
	int size = 0;
	for (int i = 0; i < instrs->s_used; i++) {
	instruction *instr = &instrs->s_instrs[i];
	if (!same_location(loc, instr->i_loc)) {
	RETURN_IF_ERROR(assemble_emit_location(a, loc, size));
	loc = instr->i_loc;
	size = 0;
	}
	size += instr_size(instr);
	}
	RETURN_IF_ERROR(assemble_emit_location(a, loc, size));
	return SUCCESS;
	}
	static void
	write_instr(_Py_CODEUNIT *codestr, instruction *instr, int ilen)
	{
	int opcode = instr->i_opcode;
	assert(!IS_PSEUDO_INSTR(opcode));
	int oparg = instr->i_oparg;
	assert(OPCODE_HAS_ARG(opcode) || oparg == 0);
	int caches = _PyOpcode_Caches[opcode];
	switch (ilen - caches) {
	case 4:
	codestr->op.code = EXTENDED_ARG;
	codestr->op.arg = (oparg >> 24) & 0xFF;
	codestr++;
	/* fall through */
	case 3:
	codestr->op.code = EXTENDED_ARG;
	codestr->op.arg = (oparg >> 16) & 0xFF;
	codestr++;
	/* fall through */
	case 2:
	codestr->op.code = EXTENDED_ARG;
	codestr->op.arg = (oparg >> 8) & 0xFF;
	codestr++;
	/* fall through */
	case 1:
	codestr->op.code = opcode;
	codestr->op.arg = oparg & 0xFF;
	codestr++;
	break;
	default:
	Py_UNREACHABLE();
	}
	while (caches--) {
	codestr->op.code = CACHE;
	codestr->op.arg = 0;
	codestr++;
	}
	}
	/* assemble_emit_instr()
	Extend the bytecode with a new instruction.
	Update lnotab if necessary.
	*/
	static int
	assemble_emit_instr(struct assembler *a, instruction *instr)
	{
	Py_ssize_t len = PyBytes_GET_SIZE(a->a_bytecode);
	_Py_CODEUNIT *code;
	int size = instr_size(instr);
	if (a->a_offset + size >= len / (int)sizeof(_Py_CODEUNIT)) {
	if (len > PY_SSIZE_T_MAX / 2) {
	return ERROR;
	}
	RETURN_IF_ERROR(_PyBytes_Resize(&a->a_bytecode, len * 2));
	}
	code = (_Py_CODEUNIT *)PyBytes_AS_STRING(a->a_bytecode) + a->a_offset;
	a->a_offset += size;
	write_instr(code, instr, size);
	return SUCCESS;
	}
	static int
	assemble_emit(struct assembler *a, instr_sequence *instrs,
	int first_lineno, PyObject *const_cache)
	{
	RETURN_IF_ERROR(assemble_init(a, first_lineno));
	for (int i = 0; i < instrs->s_used; i++) {
	instruction *instr = &instrs->s_instrs[i];
	RETURN_IF_ERROR(assemble_emit_instr(a, instr));
	}
	RETURN_IF_ERROR(assemble_location_info(a, instrs, a->a_lineno));
	RETURN_IF_ERROR(assemble_exception_table(a, instrs));
	RETURN_IF_ERROR(_PyBytes_Resize(&a->a_except_table, a->a_except_table_off));
	RETURN_IF_ERROR(_PyCompile_ConstCacheMergeOne(const_cache, &a->a_except_table));
	RETURN_IF_ERROR(_PyBytes_Resize(&a->a_linetable, a->a_location_off));
	RETURN_IF_ERROR(_PyCompile_ConstCacheMergeOne(const_cache, &a->a_linetable));
	RETURN_IF_ERROR(_PyBytes_Resize(&a->a_bytecode, a->a_offset * sizeof(_Py_CODEUNIT)));
	RETURN_IF_ERROR(_PyCompile_ConstCacheMergeOne(const_cache, &a->a_bytecode));
	return SUCCESS;
	}

That's purely internal code. I don't think we need C-API support for this, but a Python implementation would be nice. Or a static method on code objects to build the byte string from a Python list of positions. Given the complexity of the format and the risk of future changes, I think it's in CPython's responsibility to generate this format from something user friendly.

compile.h defines the location like this:

/* source location information */
typedef struct {
    int lineno;
    int end_lineno;
    int col_offset;
    int end_col_offset;
} _PyCompilerSrcLocation;

This looks like a function could help that accepts a sequence of tuples of these four values and maps it to a bytes object with the corresponding line table in it.

We'd probably still have to reimplement it in Cython in order to generate the line table string in "older" Python versions. But we could at least rely on it from CPython 3.13 on.

[scoder]

@markshannon, could you please comment whether you consider this the right approach? To me, a line table seems required to generate source level monitoring events.

[scoder]

@markshannon any comments? Given that Python 3.12 broke an entire C-API feature, I consider it a blocker for 3.13 to fix it.