Possible to update python?

[exitrip]

I'm trying to use the snap install with oss-cad-suite + litex environments and I think I'm hitting a 3.8 vs 3.11 issue with a script crashing:

Python path configuration:
  PYTHONHOME = '/home/engin/oss-cad-suite'
  PYTHONPATH = (not set)
  program name = '/snap/openxc7/x1/usr/bin/nextpnr-xilinx'
  isolated = 0
  environment = 1
  user site = 0
  import site = 1
  sys._base_executable = '/snap/openxc7/x1/usr/bin/nextpnr-xilinx'
  sys.base_prefix = '/home/engin/oss-cad-suite'
  sys.base_exec_prefix = '/home/engin/oss-cad-suite'
  sys.executable = '/snap/openxc7/x1/usr/bin/nextpnr-xilinx'
  sys.prefix = '/home/engin/oss-cad-suite'
  sys.exec_prefix = '/home/engin/oss-cad-suite'
  sys.path = [
    '/home/engin/oss-cad-suite/lib/python38.zip',
    '/home/engin/oss-cad-suite/lib/python3.8',
    '/home/engin/oss-cad-suite/lib/python3.8/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f4c50fb1c00 (most recent call first):
<no Python frame>
Traceback (most recent call last):

Can the snap python version be updated to 3.11?

[hansfbaier]

I would have to update the base image of it. Probably about time.
I would really recommend using the nix version though.

[exitrip]

Yeah, I was first trying to use the Nix deployment, but wasn't sure how to square the nix dev shell with a python venv, so switched back to the snap install. I'll try it again.

[exitrip]

So naively trying a complex install of a bunch of extra python modules (see litex_setup.py) into the nix dev shell doesn't work. No surprise as it seems Nix's whole aim is to prevent that type of thing... but a quick pass over Nix's python functionality documentation also doesn't look friendly to litex's "clone n build all the whl locally" approach either.

Surprisingly, sourcing the a venv (via a script that also monkeys with PATH) does work on top of the nix dev shell, up to the same "no module" error running write_json colorlight_i9plus.json in the yosys script.

Now this looks a lot less like a python path / versioning issue, as all the nix paths are there... but that also seems to be the only reason python would fail to find a builtin module.

5. Executing JSON backend.

Warnings: 2 unique messages, 2 total
End of script. Logfile hash: 04c9519e14, CPU: user 49.28s system 1.41s, MEM: 479.84 MB peak
Yosys 0.36 (git sha1 8f07a0d8404, gcc 12.3.0 -fPIC -Os)
Time spent: 26% 12x techmap (14 sec), 12% 32x opt_clean (6 sec), ...
Python path configuration:
  PYTHONHOME = '/home/engin/oss-cad-suite'
  PYTHONPATH = '/nix/store/pa9k6hxg9rdd29ncjmyrw4593w4zdqqa-python3.10-pyyaml-6.0.1/lib/python3.10/site-packages:/nix/store/l8d6q4y9cvp2d5qf2sxxj072668bfcjr-python3-3.10.13/lib/python3.10/site-packages:/nix/store/jpkgydqlmczm9z53m75igsxjmj64fsrk-python3.10-textx-3.0.0/lib/python3.10/site-packages:/nix/store/cqd8p76adwqfhlxzi21dsqagadlzdhl7-python3.10-arpeggio-2.0.2/lib/python3.10/site-packages:/nix/store/m1x5p772qknracqs62cn7082v0mii28i-python3.10-click-8.1.7/lib/python3.10/site-packages:/nix/store/hpfbzc5axcg7749p01g7rnbhx6mp18dx-python3.10-future-0.18.3/lib/python3.10/site-packages:/nix/store/68ihda5zzb37dd7azxaby1rlikaimd20-python3.10-setuptools-68.2.2/lib/python3.10/site-packages:/nix/store/1553c5k26qc93jm2kqdc8p3vxx1bhy5s-python3.10-simplejson-3.19.2/lib/python3.10/site-packages:/nix/store/0vjzm74r3v9zv2v0z0y375bllnd1jymm-python3.10-intervaltree-3.1.0/lib/python3.10/site-packages:/nix/store/j8ary4z5pn669w1lwacyfgb17sqdgrhm-python3.10-sortedcontainers-2.4.0/lib/python3.10/site-packages:/nix/store/byc5sqi8ci9y19jq48im5zqfcg86nmzn-prjxray-76401bd93e493fd5ff4c2af4751d12105b0f4f6d/usr/share/python3/:/nix/store/w7607ki06jzhn1wc8901kwpb2q1n1acw-python3.11-fasm/lib/python3.11/site-packages/'
  program name = 'nextpnr-xilinx'
  isolated = 0
  environment = 1
  user site = 0
  import site = 1
  sys._base_executable = '/nix/store/g3rx7i61f781j13jqsjzwb7sliyggcp8-nextpnr-xilinx-0.7.0/bin/nextpnr-xilinx'
  sys.base_prefix = '/home/engin/oss-cad-suite'
  sys.base_exec_prefix = '/home/engin/oss-cad-suite'
  sys.platlibdir = 'lib'
  sys.executable = '/nix/store/g3rx7i61f781j13jqsjzwb7sliyggcp8-nextpnr-xilinx-0.7.0/bin/nextpnr-xilinx'
  sys.prefix = '/home/engin/oss-cad-suite'
  sys.exec_prefix = '/home/engin/oss-cad-suite'
  sys.path = [
    '/nix/store/pa9k6hxg9rdd29ncjmyrw4593w4zdqqa-python3.10-pyyaml-6.0.1/lib/python3.10/site-packages',
    '/nix/store/l8d6q4y9cvp2d5qf2sxxj072668bfcjr-python3-3.10.13/lib/python3.10/site-packages',
    '/nix/store/jpkgydqlmczm9z53m75igsxjmj64fsrk-python3.10-textx-3.0.0/lib/python3.10/site-packages',
    '/nix/store/cqd8p76adwqfhlxzi21dsqagadlzdhl7-python3.10-arpeggio-2.0.2/lib/python3.10/site-packages',
    '/nix/store/m1x5p772qknracqs62cn7082v0mii28i-python3.10-click-8.1.7/lib/python3.10/site-packages',
    '/nix/store/hpfbzc5axcg7749p01g7rnbhx6mp18dx-python3.10-future-0.18.3/lib/python3.10/site-packages',
    '/nix/store/68ihda5zzb37dd7azxaby1rlikaimd20-python3.10-setuptools-68.2.2/lib/python3.10/site-packages',
    '/nix/store/1553c5k26qc93jm2kqdc8p3vxx1bhy5s-python3.10-simplejson-3.19.2/lib/python3.10/site-packages',
    '/nix/store/0vjzm74r3v9zv2v0z0y375bllnd1jymm-python3.10-intervaltree-3.1.0/lib/python3.10/site-packages',
    '/nix/store/j8ary4z5pn669w1lwacyfgb17sqdgrhm-python3.10-sortedcontainers-2.4.0/lib/python3.10/site-packages',
    '/nix/store/byc5sqi8ci9y19jq48im5zqfcg86nmzn-prjxray-76401bd93e493fd5ff4c2af4751d12105b0f4f6d/usr/share/python3/',
    '/nix/store/w7607ki06jzhn1wc8901kwpb2q1n1acw-python3.11-fasm/lib/python3.11/site-packages/',
    '/home/engin/oss-cad-suite/lib/python310.zip',
    '/home/engin/oss-cad-suite/lib/python3.10',
    '/home/engin/oss-cad-suite/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ffff4f9f7c0 (most recent call first):
  <no Python frame>
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/engin/oss-cad-suite/litex/litex-boards/litex_boards/targets/colorlight_i9plus.py", line 190, in <module>
    main()
  File "/home/engin/oss-cad-suite/litex/litex-boards/litex_boards/targets/colorlight_i9plus.py", line 179, in main
    builder.build(**parser.toolchain_argdict)
  File "/home/engin/oss-cad-suite/lib/python3.11/site-packages/litex/soc/integration/builder.py", line 411, in build
    vns = self.soc.build(build_dir=self.gateware_dir, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/engin/oss-cad-suite/lib/python3.11/site-packages/litex/soc/integration/soc.py", line 1461, in build
    return self.platform.build(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/engin/oss-cad-suite/lib/python3.11/site-packages/litex/build/xilinx/platform.py", line 98, in build
    return self.toolchain.build(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/engin/oss-cad-suite/lib/python3.11/site-packages/litex/build/xilinx/yosys_nextpnr.py", line 172, in build
    return YosysNextPNRToolchain.build(self, platform, fragment, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/engin/oss-cad-suite/lib/python3.11/site-packages/litex/build/yosys_nextpnr_toolchain.py", line 127, in build
    return GenericToolchain.build(self, platform, fragment, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/engin/oss-cad-suite/lib/python3.11/site-packages/litex/build/generic_toolchain.py", line 123, in build
    self.run_script(script)
  File "/home/engin/oss-cad-suite/lib/python3.11/site-packages/litex/build/yosys_nextpnr_toolchain.py", line 234, in run_script
    raise OSError("Error occured during Yosys/Nextpnr's script execution.")
OSError: Error occured during Yosys/Nextpnr's script execution.
⦗OSS CAD Suite⦘ [nix(openXC7)] 

Anyway, all that's to say, maybe updating the snap core et al wont actually solve my troubles using openxc7 with litex. The toolchains all work with trivial verilog examples. Cheers.

[exitrip]

okay okay, last post. If I do a VERY bad thing and cp ./lib/python3.11 ./lib/python3.10 the build progresses past this issue, and deeper magic number stuff breaks as expected

File "/home/engin/oss-cad-suite/lib/python3.10/re/_compiler.py", line 18, in <module>
    assert _sre.MAGIC == MAGIC, "SRE module mismatch"

Path of least resistance is to find a release of litex that uses 3.10 and use it with the nix-toolchain.

[hansfbaier]

Yeah, I was first trying to use the Nix deployment, but wasn't sure how to square the nix dev shell with a python venv, so switched back to the snap install. I'll try it again.

Using a python venv in a nix devshell should work.
If you have problems with that, I can help.
I probably might need to add the venv packages to the devshell, if they are missing.
Let me know how that works for you.

[exitrip]

@hansfbaier you may already be aware of this, but your snap install is straight up hardcoded into HEAD of Litex:

 if self.is_openxc7:
            prjxray_db_dir = os.environ.get('PRJXRAY_DB_DIR')
            if prjxray_db_dir is None or prjxray_db_dir == "":
                prjxray_db_dir = '/snap/openxc7/current/opt/nextpnr-xilinx/external/prjxray-db/'

in a few places in this file
https://github.com/enjoy-digital/litex/blob/master/litex/build/xilinx/yosys_nextpnr.py#L120

[hansfbaier]

Yes, but only if the environment variable is not set, and it is set in the nix toolchain.