Multiprocessing error when featurizing many band structure objects

[utf]

I've run into an issue trying to featurize a dataframe containing ~1000 band structures when using multiprocessing. I've confirmed that this is not an issue relating to any specific featurizer by creating a dummy featurizer that doesn't do anything. The issue seems to be that we are trying to pack too much data either through pickle or some other part of the multiprocessing library. I think the issue is exacerbated as I'm using uniform band structures, some of which are likely quite large.

If I run the featurization in serial mode I don't see the error.

Example code

from matminer.data_retrieval.retrieve_MP import MPDataRetrieval
from matminer.featurizers.base import BaseFeaturizer

class DummyBSFeaturizer(BaseFeaturizer):

    def featurize(self, bs):
        return 1

    def feature_labels(self):
        return ["dummy_features"]
    
mpdr = MPDataRetrieval()

criteria = {
    "band_structure": {"$exists": True},
    "elements": {"$nin": ["H"]},
    "band_gap": {"$gt": 0, "$lte": 2},
    "e_above_hull": {"$lte": 0.20},
    "nsites": {"$gte": 1, "$lte": 80},
    "icsd_ids": {"$exists": True, "$ne": []},
    "has_bandstructure": True
}

# get a dataframe of ~5000 band structures. Note, this takes a LONG time.
df = mpdr.get_dataframe(criteria=criteria, properties=['bandstructure_uniform'])

dbsf = DummyBSFeaturizer()
df = dbsf.featurize_dataframe(df[2000:3000], "bandstructure_uniform")

Error traceback

error                                     Traceback (most recent call last)
<ipython-input-55-06ffb3880a7c> in <module>
      1 dbsf = DummyBSFeaturizer()
----> 2 df = dbsf.featurize_dataframe(df[2000:3000], "bandstructure_uniform")

~/dev/src/matminer/matminer/featurizers/base.py in featurize_dataframe(self, df, col_id, ignore_errors, return_errors, inplace, multiindex, pbar)
    338                                        ignore_errors=ignore_errors,
    339                                        return_errors=return_errors,
--> 340                                        pbar=pbar)
    341 
    342         # Make sure the dataframe can handle multiindices

~/dev/src/matminer/matminer/featurizers/base.py in featurize_many(self, entries, ignore_errors, return_errors, pbar)
    465                                return_errors=return_errors,
    466                                ignore_errors=ignore_errors)
--> 467                 return p.map(func, entries, chunksize=self.chunksize)
    468 
    469     def featurize_wrapper(self, x, return_errors=False, ignore_errors=False):

~/miniconda3/envs/common_env3/lib/python3.6/multiprocessing/pool.py in map(self, func, iterable, chunksize)
    286         in a list that is returned.
    287         '''
--> 288         return self._map_async(func, iterable, mapstar, chunksize).get()
    289 
    290     def starmap(self, func, iterable, chunksize=None):

~/miniconda3/envs/common_env3/lib/python3.6/multiprocessing/pool.py in get(self, timeout)
    668             return self._value
    669         else:
--> 670             raise self._value
    671 
    672     def _set(self, i, obj):

~/miniconda3/envs/common_env3/lib/python3.6/multiprocessing/pool.py in _handle_tasks(taskqueue, put, outqueue, pool, cache)
    448                         break
    449                     try:
--> 450                         put(task)
    451                     except Exception as e:
    452                         job, idx = task[:2]

~/miniconda3/envs/common_env3/lib/python3.6/multiprocessing/connection.py in send(self, obj)
    204         self._check_closed()
    205         self._check_writable()
--> 206         self._send_bytes(_ForkingPickler.dumps(obj))
    207 
    208     def recv_bytes(self, maxlength=None):

~/miniconda3/envs/common_env3/lib/python3.6/multiprocessing/connection.py in _send_bytes(self, buf)
    391         n = len(buf)
    392         # For wire compatibility with 3.2 and lower
--> 393         header = struct.pack("!i", n)
    394         if n > 16384:
    395             # The payload is large so Nagle's algorithm won't be triggered

error: 'i' format requires -2147483648 <= number <= 2147483647

Possible fixes

Short of manually remapping the band structure numpy data to shared memory, I can't think of an easy fix for this.

As this error isn't raised when featurization is done in serial mode, perhaps a short term solution would be to set the default value of n_jobs to 1 for all band structure featurizers?

[ardunn]

related to python/cpython#10305

Does this by chance happen on python 3.8?

[utf]

Nope.

$ python --version
Python 3.6.7 :: Anaconda, Inc.

[ardunn]

yeah no i mean have you tried it on python 3.8?

[ardunn]

apparently that is when the fix I sent above was applied in

[utf]

Oh, that looks like it will fix it!

I'll give it a test.

[utf]

Ok, so this fixes the bug. However, requiring python 3.8 for matminer seems pretty steep.

What do you think about disabling multiprocessing in band structure featurizers for now? AFAIK, the band structure featurizers aren't particularly heavy anyway, so disabling multiprocessing won't have much of an effect. Especially as pickling the band structure data probably takes longer than generating the features themselves.

[ardunn]

I'd say set n_jobs conditionally based on the python version. We already do this to avoid multiprocessing problems between 2.7 and 3.0+ in BaseFeaturizer:

        else:
            if sys.version_info[0] < 3:
                warnings.warn("Multiprocessing is not supported in "
                              "matminer for Python 2.x. Multiprocessing has "
                              "been disabled. Please upgrade to Python 3.x to "
                              "enable multiprocessing.")

                self.set_n_jobs(1)

This particular piece of code should probably be removed since we don't support 2.x anymore but that's beside the point in this issue. A similar thing for versions 3.8 in Bandstructure featurizers seems like the best solution to me

[utf]

Sounds good.

[ardunn]

@utf I don't know the timing of the BS featurizers well, so if the pickling does take longer than generating the features I'd have no problem disabling it all together for BS featurizers. your call

[utf]

It really depends on the size of the band structures. For the uniform band structures I used in the code above, multiprocessing really doesn't make much difference but for smaller ones it might.