Implement PEP 3154 (pickle protocol 4)

[avassalotti]

BPO	17810
Nosy	@tim-one, @loewis, @rhettinger, @ncoghlan, @pitrou, @larryhastings, @avassalotti, @asvetlov, @serhiy-storchaka
Dependencies	bpo-15397: Unbinding of methods bpo-17893: Refactor reduce protocol implementation
Files	9f1be171da08.diff framing.patch prefetch.patch framing2.patch framing3.patch methods.patch 8434af450da0.diff framing.diff pickle_frame_headers.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/avassalotti'
closed_at = <Date 2013-11-24.04:39:41.547>
created_at = <Date 2013-04-21.06:48:58.149>
labels = ['type-feature', 'library', 'release-blocker']
title = 'Implement PEP 3154 (pickle protocol 4)'
updated_at = <Date 2013-11-25.21:56:35.101>
user = 'https://github.com/avassalotti'

bugs.python.org fields:

activity = <Date 2013-11-25.21:56:35.101>
actor = 'serhiy.storchaka'
assignee = 'alexandre.vassalotti'
closed = True
closed_date = <Date 2013-11-24.04:39:41.547>
closer = 'alexandre.vassalotti'
components = ['Library (Lib)']
creation = <Date 2013-04-21.06:48:58.149>
creator = 'alexandre.vassalotti'
dependencies = ['15397', '17893']
files = ['29966', '30068', '30072', '30094', '30118', '30213', '32640', '32709', '32840']
hgrepos = ['184']
issue_num = 17810
keywords = ['patch']
message_count = 63.0
messages = ['187496', '187500', '187510', '187512', '187516', '187830', '187833', '187834', '187874', '187876', '187877', '187878', '187891', '187896', '187918', '188087', '188089', '188090', '188096', '188098', '188102', '188109', '188227', '188280', '188281', '188282', '188312', '188315', '188320', '188327', '188330', '188331', '188338', '188351', '188363', '188889', '188971', '188989', '189017', '190554', '190583', '195499', '195522', '195583', '202939', '203339', '203420', '203435', '204066', '204067', '204088', '204089', '204093', '204097', '204175', '204176', '204389', '204390', '204391', '204397', '204399', '204401', '204426']
nosy_count = 13.0
nosy_names = ['tim.peters', 'loewis', 'rhettinger', 'ncoghlan', 'pitrou', 'larry', 'alexandre.vassalotti', 'Arfrever', 'asvetlov', 'neologix', 'python-dev', 'serhiy.storchaka', 'mstefanro']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue17810'
versions = ['Python 3.4']

[avassalotti]

I have restarted the work on PEP-3154. Stefan Mihaila had begun an implementation as part of the Google Summer of Code 2012. Unfortunately, he hit multiple roadblocks which prevented him to finish his work by the end of the summer. He previously shown interest in completing his implementation. However he got constrained by time and never resumed his work.

So I am taking over the implementation of the PEP. I have decided to go forward with a brand new code, using Stefan's work only as a guide. At the moment, I have completed about half of the PEP---missing only support for calling __new__ with keyword arguments and the use of new qualified name for referring objects.

Design-wise, there is still a few things that we should discuss. For example, I think Stefan's idea, which is not specified in the PEP, to eliminate PUT opcodes is interesting. His proposal was to emit an implicit PUT opcode after each object pickled and make the Pickler and Unpickler classes agree on the scheme. A drawback of this implicit scheme is we cannot be selective about which object we save in the memo during unpickling. That means, for example, we won't be able to make pickletools.optimize work with protocol 4 to reduce the memory footprint of the unpickling process. This scheme also alters the meaning of all previously defined opcodes because of the implicit PUTs, which is sort of okay because we are changing protocol. Alternatively, we could use an explicit scheme by defining new "fat" opcodes, for the built-in types we care about, which includes memoization. This scheme would a bit more flexible however it would also be slightly more involved implementation-wise. In any case, I will run benchmarks to see if either schemes are worthwhile.

[pitrou]

Thank you for reviving this :)
A couple of questions:

• why ADDITEM in addition to ADDITEMS? I don't think single-element sets are an important use case (as opposed to, say, single-element tuples)
• what is the purpose of STACK_GLOBAL? I would say memoization of common names but you pass memoize=False

For example, I think Stefan's idea, which is not specified in the
PEP, to eliminate PUT opcodes is interesting. His proposal was to
emit an implicit PUT opcode after each object pickled and make the
Pickler and Unpickler classes agree on the scheme.

Are the savings worth it?
I've tried pickletools.optimize() on two objects:

• a typical data dict (http.client.responses). The pickle length decreases from 1155 to 1063 (8% shrink); unpickling is faster by 4%.

• a Logger object (logging.getLogger("foobar"). The pickle length decreases from 427 to 389 (9% shrink); unpickling is faster by 2%.

[serhiy-storchaka]

Link to the previous attempt: bpo-15642.

[serhiy-storchaka]

Memoization consumes memory during pickling. For now every memoized object requires memory for:

dict's entity;
an id() integer object;
a 2-element tuple;
a pickle's index (an integer object).

It's about 80 bytes on 32-bit platform (and twice as this on 64-bit). For data which contains a lot of floats it can be cumbersome.

[pitrou]

Memoization consumes memory during pickling. For now every memoized
object requires memory for:

dict's entity;
an id() integer object;
a 2-element tuple;
a pickle's index (an integer object).

It's about 80 bytes on 32-bit platform (and twice as this on 64-bit).

As far as I understand, Alexandre doesn't propose to suppress
memoization, only to make it implicit. Therefore the memory overhead
would be the same (but the pickle would have less opcodes).

For data which contains a lot of floats it can be cumbersome.

Apparently, floats don't get memoized:

>>> pickletools.dis(pickle.dumps([1.0, 2.0]))
    0: \x80 PROTO      3
    2: ]    EMPTY_LIST
    3: q    BINPUT     0
    5: (    MARK
    6: G        BINFLOAT   1.0
   15: G        BINFLOAT   2.0
   24: e        APPENDS    (MARK at 5)
   25: .    STOP

[rhettinger]

I would like to see Proto4 include an option for compression (zlib,bz2) or somesuch and become self-decompressing upon unpickling. The primary use cases for pickling involve writing to disk or transmitting across a wire -- both use cases benefit from compression (with reduced read/write times).

[neologix]

I would like to see Proto4 include an option for compression
(zlib,bz2) or somesuch and become self-decompressing upon unpickling.

I don't see what this would bring over explicit compression:

• depending on the use case, you may want to use different compression algorithms, e.g. for disk you may want higher compression ratio like bzip2/lzma, but for wire you'd prefer something fast like snappy
• supporting multiple compression algorithms and levels would complicate the API
• this would probably complicate the code, since you'd have to support optional compression, and have a way to indicate which format is used
• that's really mixing two entirely different concepts (serialization vs compression)

[pitrou]

I don't see what this would bring over explicit compression:

• depending on the use case, you may want to use different compression algorithms, e.g. for disk you may want higher compression ratio like bzip2/lzma, but for wire you'd prefer something fast like snappy
• supporting multiple compression algorithms and levels would complicate the API
• this would probably complicate the code, since you'd have to support optional compression, and have a way to indicate which format is used
• that's really mixing two entirely different concepts (serialization vs compression)

I agree with Charles-François.
A feature that may be actually nice to have in the pickle protocol would
be some framing, to help with streaming unpickling (right now unpickling
a stream can read almost one byte at a time, IIRC).
However, that would also make the protocol and the pickler significantly
more complex.

[pitrou]

A proof of concept hack to enable framing on pickle showed a massive performance increase on streaming unpickling (up to 5x faster with a C file object such as io.BytesIO, up to 150x faster with a pure Python file object such as _pyio.BytesIO). There is a slight slowdown on non-streaming operation, but that could probably be optimized.

[pitrou]

(note: I've updated PEP-3154 with framing and GLOBAL_STACK)

[serhiy-storchaka]

A feature that may be actually nice to have in the pickle protocol would
be some framing, to help with streaming unpickling (right now unpickling
a stream can read almost one byte at a time, IIRC).
However, that would also make the protocol and the pickler significantly
more complex.

What if just use io.BufferedReader?

    if not isinstance(file, io.BufferedReader):
        file = io.BufferedReader(file)

(at start of _Unpickler.__init__)

[pitrou]

What if just use io.BufferedReader?

if not isinstance(file, io.BufferedReader):
    file = io.BufferedReader(file)

(at start of _Unpickler.__init__)

Two problems:

1. semantically, it is wrong; the BufferedReader will read bytes beyond
   the pickle end, so the underlying stream will be desynchronized

2. performance-wise, it doesn't solve the issue either: read() method
   calls are costly, even on an optimized C object