Can't deparse simple loop in Python 3.8

[laike9m]

To reproduce, run

import sys
import uncompyle6

for i in range(2):
    pass

uncompyle6.deparse_code2str(code=sys._getframe().f_code, out=sys.stdout)

which gives:

Instruction context:

   8      30  LOAD_NAME                uncompyle6
             32  LOAD_ATTR                deparse_code2str
             34  LOAD_NAME                sys
             36  LOAD_METHOD              _getframe
             38  CALL_METHOD_0         0  ''
             40  LOAD_ATTR                f_code
             42  LOAD_NAME                sys
             44  LOAD_ATTR                stdout
             46  LOAD_CONST               ('code', 'out')
->           48  CALL_FUNCTION_KW_2     2  ''
Traceback (most recent call last):
  File "/Users/laike9m/.pyenv/versions/3.8.0/envs/Cyberbrain@3.8.0/src/uncompyle6/uncompyle6/semantics/pysource.py", line 2434, in build_ast
    ast = python_parser.parse(self.p, tokens, customize)
  File "/Users/laike9m/.pyenv/versions/3.8.0/envs/Cyberbrain@3.8.0/src/uncompyle6/uncompyle6/parser.py", line 605, in parse
    ast = p.parse(tokens)
  File "/Users/laike9m/.pyenv/versions/Cyberbrain@3.8.0/lib/python3.8/site-packages/spark_parser/spark.py", line 501, in parse
    self.error(tokens, i-1)
  File "/Users/laike9m/.pyenv/versions/3.8.0/envs/Cyberbrain@3.8.0/src/uncompyle6/uncompyle6/parser.py", line 191, in error
    raise ParserError(err_token, err_token.offset)
uncompyle6.parser.ParserError: Parse error at or near `CALL_FUNCTION_KW_2' instruction at offset 48


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "uncompile.py", line 8, in <module>
    uncompyle6.deparse_code2str(code=sys._getframe().f_code, out=sys.stdout)
  File "/Users/laike9m/.pyenv/versions/3.8.0/envs/Cyberbrain@3.8.0/src/uncompyle6/uncompyle6/semantics/pysource.py", line 2582, in deparse_code2str
    return code_deparse(
  File "/Users/laike9m/.pyenv/versions/3.8.0/envs/Cyberbrain@3.8.0/src/uncompyle6/uncompyle6/semantics/pysource.py", line 2504, in code_deparse
    deparsed.ast = deparsed.build_ast(tokens, customize, isTopLevel=isTopLevel)
  File "/Users/laike9m/.pyenv/versions/3.8.0/envs/Cyberbrain@3.8.0/src/uncompyle6/uncompyle6/semantics/pysource.py", line 2437, in build_ast
    raise ParserError(e, tokens)
uncompyle6.semantics.parser_error.ParserError: --- This code section failed: ---

   1       0  LOAD_CONST               0
           2  LOAD_CONST               None
           4  IMPORT_NAME              sys
           6  STORE_NAME               sys

   2       8  LOAD_CONST               0
          10  LOAD_CONST               None
          12  IMPORT_NAME              uncompyle6
          14  STORE_NAME               uncompyle6

   5      16  LOAD_NAME                range
          18  LOAD_CONST               2
          20  CALL_FUNCTION_1       1  ''
          22  GET_ITER
          24  FOR_ITER             30  'to 30'
          26  STORE_NAME               i

   6      28  CONTINUE             24  'to 24'

   8      30  LOAD_NAME                uncompyle6
          32  LOAD_ATTR                deparse_code2str
          34  LOAD_NAME                sys
          36  LOAD_METHOD              _getframe
          38  CALL_METHOD_0         0  ''
          40  LOAD_ATTR                f_code
          42  LOAD_NAME                sys
          44  LOAD_ATTR                stdout
          46  LOAD_CONST               ('code', 'out')
          48  CALL_FUNCTION_KW_2     2  ''
          50  POP_TOP

Parse error at or near `CALL_FUNCTION_KW_2' instruction at offset 48

Python version 3.8.0, uncompyle6 version 3.5.0

[rocky]

To set expectations, there are lots of bugs in 3.8 deparsing. Python 3.8 code-generation has changed significantly since 3.7.

I know about some of these (but I haven't logged them as issues - I'm so nasty.) It's not hard to find bugs, it is a lot harder to fix them. That is why I have started to document the process I use to fix bugs. For this bug, see for example https://github.com/rocky/python-uncompyle6/wiki/Fixing-Issue-%23293-(Python-3.8-grammar-problem)

[laike9m]

That is very helpful. And I would appreciate if you can log things that are broken in 3.8, so when error happens in the future I know it's expected, which could save me a lot of time.

[rocky]

I'm now thinking that this fix is not as good as fixing the scanner to not turn this into a the fake CONTINUE instruction. The way it would know this is to see if the next instruction is the target of a FOR_ITER. This check code to modifiy is somewhere around https://github.com/rocky/python-uncompyle6/blob/master/uncompyle6/scanners/scanner3.py#L481

Are you up for making that change?

As for logging bugs, there is a difference between knowing that they exist and narrowing the problem so that it can be more easily fixed. I do realize and appreciate the extra effort that you took to narrow the problem down. The time was not "wasted" or duplicated.

To find bugs in say 3.8.0 decompilation run ./test_pyenvlib.py --3.8.0 from the test directory. Stop when you've had enough. Change 3.8.0 to some other version and you'll see that there are bugs there too.

I've never claimed this is does "perfect" decompilation as some have claimed for pycdc

[laike9m]

Thanks for the information. I can take a look when I have time, but I'd rather not letting you rely on my work, cause I've been busy developing my own project(including fixing bugs brought by 3.8). Since uncompyle6 is one of the depdency, I ocassionally fixed some small issues, but probably won't have enough time to do more than that.