Misleading `bsonjs.loads` benchmark

[oleksii-symon-corva-ai]

loads benchmark uses following logic:

    compare("bsonjs.loads(j)",
            "bson.BSON().encode(json_util.loads(j))",
            iterations,
            setup)

1. Comparing bsonjs.loads(j) to bson.BSON().encode(json_util.loads(j)) is unfair. bson.BSON().encode is supposed to encode python dicts, so we should pass it a dict directly instead of loading from json before passing.

# Before
bson.BSON().encode(json_util.loads(j))
# After
bson.BSON().encode(doc)

2. Using bson.BSON().encode is not recommeded, because of its additional performance cost, benchmark should use bson.encode method directly:

# Before
bson.BSON().encode(doc)
# After
bson.encode(doc)

Final code:

# loads
compare('bsonjs.loads(j)', 'bson.encode(doc)', iterations, setup)

Test results

If we run benchmark with fixed code we get following results:

Mac M3

Timing: bsonjs.dumps(b)
10000 loops, best of 3: 0.03492570900000001
Timing: json_util.dumps(bson.decode(b))
10000 loops, best of 3: 0.2038257080000001
bsonjs is 5.84x faster than json_util

Timing: bsonjs.loads(j)
10000 loops, best of 3: 0.03936475000000006
Timing: bson.encode(doc)
10000 loops, best of 3: 0.014574333999999967
bsonjs is 0.37x faster than json_util

Mac Intel

Timing: bsonjs.dumps(b)
10000 loops, best of 3: 0.08610098099999997
Timing: json_util.dumps(bson.decode(b))
10000 loops, best of 3: 0.6389648219999999
bsonjs is 7.42x faster than json_util

Timing: bsonjs.loads(j)
10000 loops, best of 3: 0.08243793199999949
Timing: bson.encode(doc)
10000 loops, best of 3: 0.035725252999999846
bsonjs is 0.43x faster than json_util

Summary

Looks like in reality bsonjs.loads is more than 2 times slower than bson.encode.

[ShaneHarvey]

Comparing bsonjs.loads(j) to bson.BSON().encode(json_util.loads(j)) is unfair. bson.BSON().encode is supposed to encode python dicts, so we should pass it a dict directly instead of loading from json before passing.

bsonjs.loads takes a JSON string and returns a BSON bytes. bson.encode takes a python dict and returns BSON bytes. The two methods are totally different so it does not make sense to compare the two directly. This is why we compare bsonjs.loads with the equivalent in pymongo: encode(json_util.loads(j))

Using bson.BSON().encode is not recommeded, because of its additional performance cost, benchmark should use bson.encode method directly:

Thanks for calling this out. We have not updated this library in a long time. What's more, since this library's benchmark was first published many things have changed: pymongo's bson encoding and decoding performance has improved multiple times, pymongo's json_util performance has been improved, and CPython's performance has improved as well. We should update to the latest versions and compare on Python 3.13.

[ShaneHarvey]

Updated benchmark results with pymongo 4.11 and python 3.13: (on M1 macOS)

$ python benchmark.py
Timing: bsonjs.dumps(b)
10000 loops, best of 3: 0.04682216700166464
Timing: json_util.dumps(bson.decode(b))
10000 loops, best of 3: 0.17319270805455744
bsonjs is 3.70x faster than json_util

Timing: bsonjs.loads(j)
10000 loops, best of 3: 0.053156834095716476
Timing: bson.encode(json_util.loads(j))
10000 loops, best of 3: 0.15982166700996459
bsonjs is 3.01x faster than json_util