switch back to stdlib json

[richvdh]

I found my benchmark data and it looks like stdlib json is faster than simplejson for dumping data nowadays.

Using python 3.7.5 (seconds per loop: smaller is better):

Results
=======
                             loads (large obj)   loads (small objs)    dumps (large obj)   dumps (small objs)
json 2.0.9                               0.012                0.011                0.013                0.015
simplejson 3.17.0                        0.011                0.011                0.019                0.027

[richvdh]

see also: #3009, matrix-org/python-canonicaljson#23

[auscompgeek]

Did a little experimenting, and it turns out that:

• simplejson has a feature where it'll serialise bytes by default, and
• synapse passes bytes to canonicaljson in some places.

So if canonicaljson is changed to use the stdlib json, either it'll have to decode bytestrings it gets, or synapse needs a patch to stop passing bytes to canonicaljson.

[auscompgeek]

Actually it's even worse than that: some of the C-S handlers also return bytes, I just found /voipServer does.

[richvdh]

to demonstrate the problem:

>>> import simplejson
>>> simplejson.dumps(b"abc")
'"abc"'
>>> import json
>>> json.dumps(b"abc")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'bytes' is not JSON serializable

[clokep]

I think it was mentioned in #synapse-dev:matrix.org, but I believe the first step of this was to fix the places where Synapse passes bytes into canonicaljson.

[richvdh]

Next steps on this:

• Fix known places where we pass bytes objects into json.dumps or canonicaljson.encode:
  • /voipServer is the only one I'm aware of
• Create a (pre-release?) version of canonicaljson that uses json rather than simplejson
• Update synapse to the use the new version of canonicaljson; this will probably break a number of tests
• Fix the tests
• If it all looks good, make a proper release of canonicaljson, this fixing Remove simplejson dependency python-canonicaljson#23

While we're working on canonicaljson, assuming we decide it is safe to switch to stock json, it would be good to revert the non-test parts of matrix-org/python-canonicaljson#9, which, as per the comments there, are not required with stock json.

[clokep]

Looks like we also directly use simplejson in a bunch of places in Synapse. I'll clean those up too.