function pickles are only quasi-deterministic, does this matter?

[stepleton-xx]

Pickles of functions (and pickles of things that contain functions, like classes) are not quite deterministic---they depend on iteration order of the _reverse_typemap inside dill.py. Depending on the order, either the symbol "LambdaType" or "FunctionType" will be used to represent functions. Either will work as far as unpickling goes, but having different representations of the same value can cause trouble with e.g. caching.

While most invocations of Python 2.x yield the same iteration order for _reverse_typemap, use of the -R flag (recommended for user-facing services; c.f. http://www.ocert.org/advisories/ocert-2011-003.html) randomizes this order.

Note that the functionality of -R is on by default for versions >= 3.3:
http://docs.python.org/3/whatsnew/3.3.html

[mmckerns]

@stepleton: On my most recent commit, I did notice that the ordering of the _typemap is different in 3.x [where x = 3, and possibly 2] versus older versions, including the entire 2.x line. However, in some limited testing, I didn't experience any failures to pick up LambdaType versus FunctionType, for example. I didn't however stress test it to make sure it absolutely wasn't a problem. _reverse_typemap as is, should include both LambdaType and FunctionType (and the other three cases where there are two identical types). _typemap has the types themselves as the keys, so there will be 4 less items in _typemap than in _reverse_typemap. What does that mean? Does it cover the issue?

@register can only register the type, so it's equivalent to @register(FunctionType) or @register(LambdaType), I believe. The pickled object shouldn't know what type it is, so as long as either LambdaType or FunctionType (reverse type)map to the same type... the lookup using the object as the key should uncover one of the two (LambdaType or FunctionType), depending on the possibly random order the dict was populated with.

That was my thinking in it being declared "not a problem".

Do you have any test cases for dill actually failing here? Not that the _typemap is deterministic, because, as you say, it is quasi-deterministic in +3.3... but a case were dill fails to correctly dump/load a type that it should know. Something like, save a "lambda" to a (session) file, then try loading it 1000 times -- if it fails once or more, then there's a problem. Is this something like the case you had in mind?

[asmeurer]

You can make the hash randomization deterministic for testing purposes by setting the PYTHONHASHSEED environment variable to a positive integer.

[davips]

Any news regarding providing determinism for dill? I am using it only for hashing, so is there any workaround?

[anivegesana]

Are you using Python 3.7+? Because dictionaries are now insertion order, for any particular version of Python after 3.6, the type dictionary should be deterministic, and this issue no longer applies. Since dill 0.3.6 will no longer support Python 2.7, this issue should no longer be relevant if I am not mistaken.

[davips]

Thanks for the follow-up. It still gives different hashes in python 3.8.10, despite being deterministic if we start a new execution:

In [3]: import dill as d; from hashlib import md5; f = lambda x: x + 1; dum = d.dumps(f); md5(dum).hexdigest()
Out[3]: 'bbe58ab30aec9b1feb80cd3e4b2b7024'

In [4]: import dill as d; from hashlib import md5; f = lambda x: x + 1; dum = d.dumps(f); md5(dum).hexdigest()
Out[4]: '3fc384d6edf117061f781113a60fd053'

[mmckerns]

I'm not seeing the same behavior in 3.8.13.

Python 3.8.13 (default, May 10 2022, 11:26:38) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> dill.settings['recurse'] = True
>>> from hashlib import md5
>>> f = lambda x:x+1
>>> dum = dill.dumps(f)
>>> md5(dum).hexdigest()
'65e4cf11903c322b610b68ddb7c71283'
>>> f = lambda x:x+1
>>> dum = dill.dumps(f)
>>> md5(dum).hexdigest()
'65e4cf11903c322b610b68ddb7c71283'
>>> 
>>> dill.settings['recurse'] = False
>>> f = lambda x:x+1
>>> dum = dill.dumps(f)
>>> md5(dum).hexdigest()
'a882bb1cb887cee113682e6a827455c0'
>>> f = lambda x:x+1
>>> dum = dill.dumps(f)
>>> md5(dum).hexdigest()
'a882bb1cb887cee113682e6a827455c0'

[davips]

The difference in our outputs is not due to the python version, but due to how different consoles handle the context (it seems).
If I use python instead of ipython it works, but if I put everything inside the same file and run as a script, it doesn't work anymore.
I've seen somewhere in the past about a global dict that stores the module functions, and that changes the content of what is pickled.
AFAICR, the problem is the same if you have a file with other functions defined, not necessarily a repetition.
E.g.:
File A:
def f()
def h()

File B:
def g()
def h()

Python 3.8.10 (default, Mar 15 2022, 12:22:08) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill; from hashlib import md5; dill.settings['recurse'] = True; f = lambda x:x+1; dum = dill.dumps(f); dill.settings['recurse'] = False; f = lambda x:x+1; dum = dill.dumps(f); md5(dum).hexdigest()
'a882bb1cb887cee113682e6a827455c0'
>>> import dill; from hashlib import md5; dill.settings['recurse'] = True; f = lambda x:x+1; dum = dill.dumps(f); dill.settings['recurse'] = False; f = lambda x:x+1; dum = dill.dumps(f); md5(dum).hexdigest()
'a882bb1cb887cee113682e6a827455c0'
>>> 

[mmckerns]

I see the behavior in IPython

Python 3.8.13 (default, May 10 2022, 11:26:38) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.2.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import dill

In [2]: from hashlib import md5

In [3]: f = lambda x:x+1

In [4]: dum = dill.dumps(f)

In [5]: md5(dum).hexdigest()
Out[5]: '2a7b752183b27084ae5b3806d6b6fae3'

In [6]: f = lambda x:x+1

In [7]: dum = dill.dumps(f)

In [8]: md5(dum).hexdigest()
Out[8]: 'f5b16c000549792da22973e74cf4dd3a'

[mmckerns]

Looks like IPython is doing something to modify the function, or one of it's dependencies (like globals), so I agree, it's probably a contextual issue.

[anivegesana]

A couple of other Python libraries (like huggingface datasets) depend on persistent hashing of dill pickles, so it is something that would effect at least one popular Python library in production.

What huggingface does is sort the names of the globals dictionary and it mostly works for their purposes. I am planning to rewrite globals in #466, so I am trying to make it easy to add custom processing to global dictionaries outside of dill. And it would, at minimum, benefit the guys over at huggingface.

As huggingface noticed, another big thing holding back the determinism of dill pickles are the line numbers, filename, and other environmental information stored in the code objects. We could add a setting to dill to turn these features off and sort the globals by name.

[anivegesana]

@davips For a temporary fix if you are still using dill 0.3.4 (they haven't updated the code for dill 0.3.5 yet), you can just snag the code from huggingface datasets (either import their pickler or just copy the code out of their repo.) I will be creating a more sustainable solution that is built into dill.

https://github.com/huggingface/datasets/blob/95193ae61e92aa537d0c65d37a1fd9d2393aae89/src/datasets/utils/py_utils.py#L416-L702

!pip install datasets
from hashlib import md5
import io

import dill
from datasets.utils.py_utils import Pickler

def dumps(obj, **kwds):
    file = io.BytesIO()
    Pickler(file, **kwds).dump(obj)
    return file.getvalue()

>>> md5(dill.dumps(lambda: 'q')).hexdigest()
d6cd2da806a4384c10bbea6a2c496b3c

>>> md5(dill.dumps(lambda: 'q')).hexdigest()
1d5a198b1ffbe2c3e3118c0ff6a6eb3b

>>> md5(dumps(lambda: 'q')).hexdigest()
211cbff14077367c0c870d82a56f94bb

>>> md5(dumps(lambda: 'q')).hexdigest()
211cbff14077367c0c870d82a56f94bb

[davips]

Thanks very much @anivegesana ! It seems to work fine this way and should enable resuming some interesting projects here... 🥇

[anivegesana]

As far as I am aware, sets (and frozensets) are nondeterministic but are also unsortable in some cases:

>>> sorted({object(), object()})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'object' and 'object'

The best that can be done is to either continue to produce nondeterministic pickles only when the objects in the set are incomparable or to produce nondeterministic pickles for sets no matter what.