Partial Mode generating unexpected keys

[luizbezerra-pinheiro]

After upgrading from version 0.7.1 to version 0.8.0, after some struggle spotting from where did my error come from, I found out that the partial mode wasn't working as expected.

How to replicate the bug:

1. Upgrade the package
   pip install jiter==0.8.0

2. Get a partial and incomplete JSON and try to parse it using jiter:

>>> from jiter import from_json
>>> from_json(b'{\n "mykeyname": "hello there", \n', partial_mode=True)
{'__builtim': 'ÈîPÙªª\x00\x00hel'}

After downgrading to version 0.7.1:

>>> from jiter import from_json
>>> from_json(b'{\n "mykeyname": "hello there", \n', partial_mode=True)
{'mykeyname': 'hello there'}

System information:
Python 3.12.7 (main, Oct 29 2024, 14:29:28) [GCC 11.4.0] on linux

[davidhewitt]

Thanks for the report. Looks potentially serious, x86_64 architecture?

[davidhewitt]

After a quick check on both macOS and linux, I cannot reproduce this. Are you able to share any more information about how you set up your environment / install Python, etc?

$ docker run -it python:3.12.7 /bin/bash
Unable to find image 'python:3.12.7' locally
3.12.7: Pulling from library/python
b2b31b28ee3c: Pull complete 
c3cc7b6f0473: Pull complete 
2112e5e7c3ff: Pull complete 
af247aac0764: Pull complete 
c7e62dc73b01: Pull complete 
35c3ce70330a: Pull complete 
f8af1611cc8d: Pull complete 
Digest: sha256:f71437b2bad6af0615875c8f7fbeeeae1b73e3c76b82056d283644aca5afe355
Status: Downloaded newer image for python:3.12.7
root@a54fc0f7bd9b:/# pip install jiter
Collecting jiter
  Downloading jiter-0.8.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.2 kB)
Downloading jiter-0.8.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (341 kB)
Installing collected packages: jiter
Successfully installed jiter-0.8.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: pip install --upgrade pip
root@a54fc0f7bd9b:/# python
Python 3.12.7 (main, Nov 12 2024, 06:04:35) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import jiter
>>> from jiter import from_json
>>> from_json(b'{\n "mykeyname": "hello there", \n', partial_mode=True)
{'mykeyname': 'hello there'}

[davidhewitt]

Ah, I can reproduce using aarch64 builds. macOS host running the arm docker image:

$ docker run -it python:3.12 /bin/sh
# python 
Python 3.12.7 (main, Nov 13 2024, 15:07:43) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
# pip install jiter
Collecting jiter
  Downloading jiter-0.8.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (5.2 kB)
Downloading jiter-0.8.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (333 kB)
Installing collected packages: jiter
Successfully installed jiter-0.8.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: pip install --upgrade pip
# python 
Python 3.12.7 (main, Nov 13 2024, 15:07:43) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import jiter
>>> jiter.from_json(b'{\n "mykeyname": "hello there", \n', partial_mode=True)
{'__builtim': '8 \x1b骪\x00\x00hel'}

[davidhewitt]

... but I'm now struggling to reproduce from a locally-built jiter on the same docker image. 😖

[luizbezerra-pinheiro]

Ah, I can reproduce using aarch64 builds. macOS host running the arm docker image:

$ docker run -it python:3.12 /bin/sh
# python 
Python 3.12.7 (main, Nov 13 2024, 15:07:43) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
# pip install jiter
Collecting jiter
  Downloading jiter-0.8.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (5.2 kB)
Downloading jiter-0.8.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (333 kB)
Installing collected packages: jiter
Successfully installed jiter-0.8.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: pip install --upgrade pip
# python 
Python 3.12.7 (main, Nov 13 2024, 15:07:43) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import jiter
>>> jiter.from_json(b'{\n "mykeyname": "hello there", \n', partial_mode=True)
{'__builtim': '8 \x1b骪\x00\x00hel'}

That is my setup! I use a devcontainer (linux) under MacOS (m3). At least it would break only locally, not on production

[davidhewitt]

Fixed in #175, will release 0.8.1 asap