'ImportError: No module named gcloud' error with Dataproc and Pyspark

[sudeepag]

When I try to run a Pyspark job on a Cloud Dataproc cluster, I get the following error:

ImportError: No module named gcloud

I have gcloud installed on all the nodes in the cluster (the master, as well as the worker nodes). Here are the version numbers -

Google Cloud SDK 111.0.0
bq 2.0.24
bq-nix 2.0.24
core 2016.05.20
core-nix 2016.05.05
gcloud
gsutil 4.19
gsutil-nix 4.19

However, when I submit the job after I ssh into the master node and run $ spark-submit filename.py, it runs perfectly fine.

[dhermes]

On the machine where the ImportError occurs you should

try:
    import gcloud
except ImportError
    import sys
    print(sys.path)

and sleuth around to find out why it isn't being picked up.

There's not much we can do, it sounds like a packaging issue on your machine / platform, which may be a Python issue, but isn't really anything to do with our library.

[MrTheDave]

@sudeepag did you manage to figure out what was wrong?
I'm experiencing a similar issue. I have an init script for my dataproc cluster that does a pip install gcloud but it seems to be ignoring it.

I have to manually ssh into the master and the slaves and run pip install gcloudto get it working

[sudeepag]

@MRSNARK this turned out to be a config issue rather than a gcloud issue. I don't remember exactly how I managed to fix it, but I would confirm that python is configured properly. One good sanity check would be to run python in the master and workers and ensure that you can do an import gcloud.
(which is essentially what @dhermes suggested)