This is a broad issue that I'll use to document some of the problems that have come up when scaling out the UKB pipeline.

For convenience, here is a list of issues that have been filed as a result of these problems:

• Google Storage download retry predicate additional exception snakemake/snakemake#785
• HttpError with no message halfway through large GS write workload fsspec/gcsfs#316
• Broken pipe error in long-lived write-only workload fsspec/gcsfs#315
• Debug snakemake file download failure #25
• GCP broken pipe error dask/dask-cloudprovider#219
• Adaptive cluster on GP does not add workers dask/dask-cloudprovider#218
• Unexpected ParseError fsspec/gcsfs#318

Some of these are discussed more below. Others were filed directly in upstream repos.