test_rasterio_vrt_network is failing in continuous integration tests

[shoyer]

    @network
    def test_rasterio_vrt_network(self):
        import rasterio
    
        url = 'https://storage.googleapis.com/\
        gcp-public-data-landsat/LC08/01/047/027/\
        LC08_L1TP_047027_20130421_20170310_01_T1/\
        LC08_L1TP_047027_20130421_20170310_01_T1_B4.TIF'
        env = rasterio.Env(GDAL_DISABLE_READDIR_ON_OPEN='EMPTY_DIR',
                           CPL_VSIL_CURL_USE_HEAD=False,
                           CPL_VSIL_CURL_ALLOWED_EXTENSIONS='TIF')
        with env:
>           with rasterio.open(url) as src:

xarray/tests/test_backends.py:3734: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/share/miniconda/envs/test_env/lib/python3.6/site-packages/rasterio/env.py:430: in wrapper
    return f(*args, **kwds)
/usr/share/miniconda/envs/test_env/lib/python3.6/site-packages/rasterio/__init__.py:216: in open
    s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   rasterio.errors.RasterioIOError: HTTP response code: 400 - Failed writing header

https://dev.azure.com/xarray/xarray/_build/results?buildId=150&view=ms.vss-test-web.build-test-results-tab&runId=2358&resultId=101228&paneView=debug

I'm not sure what's going on here -- the tiff file is still available at the given URL.

@scottyhq any idea?

[scottyhq]

Strange, the same test works on my laptop current packages. Some ideas below:

• in the rasterio.Envenvironment we can add CPL_CURL_VERBOSE=True to get more information about the error, but would need to capture output that typically goes to the terminal (I've done this before with https://github.com/minrk/wurlitzer).

• I'm inclined to remove CPL_VSIL_CURL_USE_HEAD=False which is not documented anywhere obvious (https://lists.osgeo.org/pipermail/gdal-dev/2014-August/039924.html) and could be behind the error message. All rasterio/gdal environment config options are detailed here https://trac.osgeo.org/gdal/wiki/ConfigOptions

• we might also consider a different URL. maybe add a tif file to https://github.com/pydata/xarray-data? or use one of same URLs that the rasterio library is using for network tif tests?
  https://github.com/mapbox/rasterio/blob/c2a495d536db6328e018cb65ce7cc0b8c559a937/tests/test_env.py#L34-L36

[shoyer]

I'd be up for any of these -- any interest in putting together a PR? Ideally we'd have a TIF that is smaller than 50 MB -- that seems like an unnecessary large amount of data to download for tests.

[scottyhq]

sounds good. i'll try to get to it this week.