Cloud Storage downloading file fails within child process

[kierkegaard13]

OS: macOS 10.12.6
Affected google-cloud-storage versions: 1.1.1 and 1.3.2
Python version: 3.6.2 (conda venv)

For v1.1.1:
Using Python's multiprocessing library to spawn a child process (via Process), the google-cloud-storage library seems to fail when calling blobl.download_to_filename(...). I don't get any error message, but the exitcode of the process is -11.

For v1.3.2
Under the same conditions, storage_client.get_bucket(bucket_name) seems to fail without an error and the process has the same exitcode of -11.

The process I'm spawning isn't daemonic. Let me know if there's any additional info I can provide. I can't really give any code snippets, and I haven't yet been able to extract any additional traceback/error info.

[dhermes]

@kierkegaard13 Thanks for filing. I'm not sure there's much we can do about google-cloud-storage==1.1.1: if the bug isn't present in later versions then it's been "fixed". Is there a particular reason you are pinning to an older version? (The latest is 1.5.0 currently.)

Could you provide more concrete code snippets so that I can try to reproduce?

[kierkegaard13]

I understand. The latest version I was able to get with pip by default was 1.3.2 (but I'll check 1.5.0 and report back), and I encountered the bug with that version as described above. Here's the load function I'm using to load the file:

def load(self, source_path: str, source_type=SourceType.File):                                                                                                                                      
    """Downloads a blob from the bucket."""                                                                                                                                                         
     print('Loading {}'.format(source_path))                                                                                                                                                         
     storage_client = storage.Client(self.project)                                                                                                                                                   
     print('getting bucket {}'.format(self.bucket_name))                                                                                                                                             
     bucket = storage_client.get_bucket(self.bucket_name)                                                                                                                                            
     print('getting blob')                                                                                                                                                                           
     blob = bucket.blob(source_path)                                                                                                                                                                 
                                                                                                                                                                                                    
     if source_type == SourceType.File:                                                                                                                                                              
        suffix = '.{}'.format(source_path.split('.')[-1]) if '.' in source_path else ''                                                                                                             
        destination_file_name = tempfile.NamedTemporaryFile(suffix=suffix).name                                                                                                                     
        blob.download_to_filename(destination_file_name)                                                                                                                                            
        LOGGER.info('Blob %s downloaded to %s.', source_path, destination_file_name)                                                                                                                
        return destination_file_name                                                                                                                                                                
     else:                                                                                                                                                                                                          
        LOGGER.info('Blob %s downloaded as string.', source_path)                                                                                                                                   
        return blob.download_as_string() 

which fails before printing 'getting blob' so assumably on the call to get_bucket. I'm using:

job_proc = Process(target=job_func, kwargs=job['Kwargs'])

to create the child process. job_func makes a call to the load function described above.

[kierkegaard13]

I was able to replicate the issue with 1.3.2 on version 1.5.0.

[dhermes]

Looks like you could benefit from os.path.splitext when computing your suffix? I'll try to replicate your issue but you may have too many missing steps in between job_func and the instance method load() described above.

[kierkegaard13]

Thanks for the tip. There's only two lines of code between the two snippets I provided, one of which is the initialization of the class the load function is attached to which sets the project and bucket name.

[dhermes]

I made a "minimally reproducible" script to try to reproduce this:

# repro_4293.py
from __future__ import print_function

import multiprocessing
import os
import sys

from google.cloud import storage


CONTENT = b'Zooooooooooooooooooooooom.'
BUCKET = u'' SYNTAX-ERROR-EEEK!
BLOB_NAME = u'foo.txt'


def get_local_blob():
    client = storage.Client()
    bucket = client.bucket(BUCKET)
    return bucket.blob(BLOB_NAME)


def populate_blob():
    blob = get_local_blob()
    if blob.exists():
        print('Blob already exists.', file=sys.stderr)
        sys.exit(1)
    blob.upload_from_string(CONTENT)


def get_contents(filename):
    current_process = multiprocessing.current_process()
    print('current_process: {}'.format(current_process))
    blob = get_local_blob()
    blob.download_to_filename(filename)


def get_filename(base_name):
    curr_dir = os.path.dirname(__file__)
    result = os.path.join(curr_dir, base_name)
    return os.path.abspath(result)


def verify_contents(filename):
    with open(filename, 'rb') as file_obj:
        contents = file_obj.read()

    name = os.path.basename(filename)
    same = contents == CONTENT
    msg = '{} as expected? {}'.format(name, same)
    print(msg)


def clean_up(*filenames):
    for filename in filenames:
        os.remove(filename)

    blob = get_local_blob()
    blob.delete()


def main():
    # 1. What version?
    print('storage.__version__: {}'.format(storage.__version__))

    # 2. Populate the blob in the bucket.
    populate_blob()

    # 3. Get two filenames (for current process and subprocess).
    filename1 = get_filename('here1.txt')
    filename2 = get_filename('here2.txt')
    if os.path.exists(filename1) or os.path.exists(filename2):
        print('File exists.', file=sys.stderr)
        sys.exit(1)

    # 4. Download to a file in current process.
    get_contents(filename1)

    # 5. Download to a file in a subprocess.
    process = multiprocessing.Process(
        target=get_contents, args=(filename2,))
    process.start()
    process.join()

    # 6. Make sure we downloaded what we expected.
    verify_contents(filename1)
    verify_contents(filename2)

    # 7. Clean up the created blob and files.
    clean_up(filename1, filename2)


if __name__ == '__main__':
    main()

and I cannot reproduce it:

$ .nox/unit-3-6/bin/python repro_4293.py
storage.__version__: 1.6.1.dev1
current_process: <_MainProcess(MainProcess, started)>
current_process: <Process(Process-1, started)>
here1.txt as expected? True
here2.txt as expected? True

[kierkegaard13]

Hm.. I guess it's something specific to my code then. I actually found a way around this issue by not using processes, so I think it's probably okay to close this unless someone else runs into it. Thanks for your help.

[dhermes]

Thanks a lot. Sorry we couldn't drill down to the root of the problem.

Let me know if there is anything else we can do.