tests fail with python 3.9

[keszybz]

+ /usr/bin/python3 -m enjarify.runtests
Traceback (most recent call last):
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writeclass.py", line 116, in toClassFile
    pool, rest_stream = classFileAfterPool(cls, opts=opts)
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writeclass.py", line 100, in classFileAfterPool
    writeMethods(pool, stream, cls.data.methods, opts=opts)
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writeclass.py", line 65, in writeMethods
    code_attrs = writebytecode.finishCodeAttrs(pool, code_irs, opts=opts)
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writebytecode.py", line 75, in finishCodeAttrs
    return {irdata.method: writeCodeAttributeTail(pool, irdata, opts=opts) for irdata in code_irs}
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writebytecode.py", line 75, in <dictcomp>
    return {irdata.method: writeCodeAttributeTail(pool, irdata, opts=opts) for irdata in code_irs}
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writebytecode.py", line 99, in writeCodeAttributeTail
    raise error.ClassfileLimitExceeded()
enjarify.jvm.error.ClassfileLimitExceeded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/runtests.py", line 45, in <module>
    executeTest('test{}'.format(i), opts)
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/runtests.py", line 30, in executeTest
    classes, errors = translate(rawdex, opts=opts, allowErrors=False)
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/main.py", line 38, in translate
    class_data = writeclass.toClassFile(cls, opts)
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writeclass.py", line 119, in toClassFile
    pool, rest_stream = classFileAfterPool(cls, opts=options.ALL)
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writeclass.py", line 100, in classFileAfterPool
    writeMethods(pool, stream, cls.data.methods, opts=opts)
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writeclass.py", line 64, in writeMethods
    code_irs.append(writebytecode.getCodeIR(pool, method, opts=opts))
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writebytecode.py", line 21, in getCodeIR
    irdata = writeir.writeBytecode(pool, method, opts)
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writeir.py", line 597, in writeBytecode
    VISIT_FUNCS[instr.type](method, dex, instr_d, type_data, block, instr)
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writeir.py", line 380, in visitFillArrayData
    block.fillarraydata(_arrStoreOps.get(elet, AASTORE), st, arrdata)
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writeir.py", line 146, in fillarraydata
    self.fillarraysub(op, [partial(self.const, val, stype) for val in vals])
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writeir.py", line 129, in fillarraysub
    cb()
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/writeir.py", line 118, in const
    self.add(ir.PrimConstant(stype, val, pool=pool))
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/ir.py", line 82, in __init__
    self.bytecode = calc.calc(st, val)
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/constants/calc.py", line 179, in calc
    return calcDouble(val)
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/constants/calc.py", line 162, in calcDouble
    def calcDouble(x): return _calcDouble(normalizeDouble(x))
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/constants/calc.py", line 157, in _calcDouble
    return _calcLong(mantissa) + bytes([L2D]) + exponent_parts
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/constants/calc.py", line 66, in _calcLong
    result += _calcInt(low) + bytes([I2L, LXOR])
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/constants/calc.py", line 50, in _calcInt
    return _calcInt(high) + _calcInt(16) + bytes([ISHL]) + _calcInt(low) + bytes([IXOR])
  File "/var/tmp/enjarify/enjarify-1.0.3/enjarify/jvm/constants/calc.py", line 47, in _calcInt
    assert high
AssertionError

This is in Fedora rawhide with java-11-openjdk-headless-11.0.8.10-1.fc33.x86_64.

[keszybz]

https://bugzilla.redhat.com/show_bug.cgi?id=1808128

[Storyyeller]

Do you know if there were any changes in Python 3.9 that could have caused this? Did it work in Python 3.8? My guess is that it's due to a change in floating point behavior, but I haven't kept up with the Python 3.9 changelog.

I just checked runtests, and the Java is only used to execute the translated files and compare the results, so Java version should be irrelevant here.

[Storyyeller]

I'd recommend adding some print statements to jvm/constants/calc.py, then diffing the console output when running tests under Python 3.8 and 3.9. Enjarify should be deterministic, so any difference in output will help narrow down what caused the change.

[keszybz]

Indeed, this was reported when Fedora transitioned to python3.9, but I somehow jumped to the conclusion that this is related to the Java version bump that happened soon after that.

Did it work in Python 3.8?

Yes.

[keszybz]

I narrowed down the failure to the following call:

$ cd enjarify/jvm/constants/
$ PYTHONPATH=. python3.8 -c 'import lookup; print(lookup.INTS[32765])'
b'\x11\x7f\xfd'
$ PYTHONPATH=. python3.9 -c 'import lookup; print(lookup.INTS[32765])'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
KeyError: 32765

The key should be there, 0x7ffd == 32765 is the only entry missing from INTS.

This is with python3.8-3.8.5-2.fc33.x86_64 and python3-3.9.0~b4-1.fc33.x86_64.

[Storyyeller]

Just to clarify, this is after running genlookup.py on 3.9 to regenerate the lookup table, right? Were there any other differences in the generated lookup.py file under 3.8 and 3.9?

[hroncok]

python 3.9 trims the dict at 65535 elements. Let me search whether this is an intentional new behavior (I don't think so).

[Storyyeller]

This sounds like a bug in Python.

[hroncok]

Indeed it does.

[hroncok]

It seems to only trims it only for literal dicts.

[hroncok]

Not entirely true, it was my assumption based on trimming the dict a bit. If I keep it full, it says:

$ python3.9 -c 'import lookup; print(len(lookup.INTS))'
65670
$ python3.8 -c 'import lookup; print(len(lookup.INTS))'
65671

[hroncok]

Reporting the bug now...

[hroncok]

https://bugs.python.org/issue41531

[hroncok]

This has been fixed in Python 3.9.0rc2.