bpo-38115: fix invalid lnotab after optimization

[Yhg1s]

Fix bpo-38115, a bug in the peephole optimizer that caused lnotab to have invalid bytecode offsets when the optimizer got rid of the last line(s) of a function. Also fix a bug where the optimizer bails out of optimization but leaves the lnotab modified. This is a very old bug, but before 3.8 the optimizer is more likely to successfully optimize away the last line of a function.

These don't change the bytecode that's generated, but they do change the code objects (the co_lnotab attribute), so though they theoretically affect Python 3.7 and earlier, we shouldn't backport this beyond 3.8. (Also, if you want to verify this manually, don't forget to touch the .py file or delete the .pyc file... not that I'm speaking of personal experience or anything.)

https://bugs.python.org/issue38115

[Yhg1s]

@markshannon might also be interested in this, I guess ;-P

[Yhg1s]

Looks like the new assertion is getting triggered by code in test_scope, I'm digging into it.

[Yhg1s]

The assert didn't consider empty bytecode, that's now fixed.

[Yhg1s]

After some consideration, this is the wrong fix. See the test_pdb failure; replacing entries for past the end to the end makes consumers of lnotab think the last opcode of the function is part of the removed lines. Instead, what we should do is truncate lnotab. Unfortunately we can't do that, because PyCode_Optimize() is passed the lnotab as a PyBytes object, and can't resize it (because that may move it, which means the caller's object is invalid.)

Considering the peephole optimizer is likely to go away in 3.9, and this bug isn't that harmful (it doesn't confuse pdb or coverage), we should probably just document that the lnotab can contain invalid bytecode offsets and ignore it for now.

[pablogsal]

Thanks @Yhg1s for helping me and for trying to solve this. I will update #14068 as we talked.