larch.pickle - A faster and secure python pickle replacement

This module can be used as transparent replacement for pickle.

Difference to original python pickle:

• no memo attribute.
• no clear_memo().
• no persistent_id interface.
• byte strings are always assumed to be utf-8 encoded.
• Pickler has an additional with_refs parameter. Setting with_refs to false, the pickler will ignore object references. This can result in an extra speed boost.

Installation

larch-pickle needs the boost library for compilation. If boost is not in the standard include path install it with:

python build_ext -I /path/to/boost install

Security

Version 1.4.0 introduced a secure parameter in Unpickler. With secure=True the Unpickler loads only objects that, are registered as secure. To register secure objects you can

• use the decorator secure_unpickle
• set the attribute __pickle_secure__ = True
• add a module name to secure_modules

Speed compared to some other pickler packages

dump Dictionaries (10 loops)

Package	Seconds	Size
larch-pickle	0.291857	9356357
marshal	0.344505	15975952
msgpack	0.368878	9810043
ujson	0.550575	13101307
json	0.815204	14365311
Pickle-3.12.2	1.16181	10276493

load Dictionaries (10 loops)

Package	Seconds
larch-pickle	0.719275
marshal	0.767289
msgpack	0.817542
Pickle-3.12.2	0.847736
json	0.91282
ujson	0.930489

dump Objects (10 loops)

Package	Seconds	Size
larch-pickle	0.337152	9656364
Pickle-3.12.2	1.15342	10416542

load Objects (10 loops)

Package	Seconds
larch-pickle	0.794334
Pickle-3.12.2	0.820135

dump Strings (10 loops)

Package	Seconds	Size
marshal	0.215026	32481517
msgpack	0.251937	28782143
larch-pickle	0.418834	10885236
json	0.513742	31701248
ujson	0.537478	30722275
Pickle-3.12.2	0.633869	17726498

load Strings (10 loops)

Package	Seconds
larch-pickle	0.206729
Pickle-3.12.2	0.253017
marshal	0.359512
msgpack	0.360152
json	0.527622
ujson	0.673696