Error "'HTTPResponse' object has no attribute 'chunked'" when running cwltool

[muffato]

Hi there,
I got this error on a fresh install of cwltool + the workflows

$ cwltool --debug workflows/tools/picard-MergeSamFiles.cwl
/home/matthieu/.local/bin/cwltool 1.0.20161207161158
Resolved 'workflows/tools/picard-MergeSamFiles.cwl' to 'file:///home/matthieu/workspace/hive/cwl/workflows/tools/picard-MergeSamFiles.cwl'
Tool definition failed initialization:
('http://dublincore.org/2012/06/14/dcterms.rdf', AttributeError("'HTTPResponse' object has no attribute 'chunked'",))
Traceback (most recent call last):
  File "/home/matthieu/.local/lib/python2.7/site-packages/cwltool/main.py", line 638, in main
    fetcher_constructor=fetcher_constructor)
  File "/home/matthieu/.local/lib/python2.7/site-packages/cwltool/load_tool.py", line 164, in validate_document
    processobj, metadata = document_loader.resolve_all(workflowobj, fileuri)
  File "/home/matthieu/.local/lib/python2.7/site-packages/schema_salad/ref_resolver.py", line 628, in resolve_all
    newctx.add_schemas(document[u"$schemas"], file_base)
  File "/home/matthieu/.local/lib/python2.7/site-packages/schema_salad/ref_resolver.py", line 269, in add_schemas
    content = self.fetch_text(fetchurl)
  File "/home/matthieu/.local/lib/python2.7/site-packages/schema_salad/ref_resolver.py", line 104, in fetch_text
    raise RuntimeError(url, e)
RuntimeError: ('http://dublincore.org/2012/06/14/dcterms.rdf', AttributeError("'HTTPResponse' object has no attribute 'chunked'",))

I have a working internet connection and can download the file with wget but also from within python

$ python
Python 2.7.6 (default, Oct 26 2016, 20:30:19) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib
>>> h = urllib.urlopen('http://dublincore.org/2012/06/14/dcterms.rdf')
>>> '<rdf' in h.read()
True

[muffato]

Just realized that the download is actually done with requests, not urllib. I've got requests 2.2.1 installed (default on ubuntu 14.04) which is higher than 1.0 https://github.com/common-workflow-language/cwltool/blob/master/setup.py#L46

[tetron]

Thanks, I haven't seen this issue. We use an additional extensions for requests called CacheControl, it is possible that is causing trouble. I'll have to investigate.

[muffato]

It was working fine yesterday and this morning until it suddenly started complaining. I haven't done any conscious upgrade of my distribution / packages. Could it be that the cache has been corrupted ?

[muffato]

I've had to upgrade requests for another tool, and it ended up fixing cwltool as well.
I suppose it's related to the bugs they've fixed yesterday: https://pypi.python.org/pypi/requests/#release-history
I'll close the issue

[mr-c]

Thank you @muffato for the update -- I'm sorry this gave you such problems.

From the requests release notes:

"Fixed regression from 2.12.2 where non-string types were rejected in the basic auth parameters. While support for this behaviour has been readded, the behaviour is deprecated and will be removed in the future."

@tetron does this mean we need to change our usage?

[tetron]

I have no idea, our use of requests is extremely simple except for the addition of the CacheControl wrapper. So if anything is using it wrong, it is more likely to be CacheControl. It's also possible we need to require a more recent version of requests than 1.0, that will require a bit more research.

[evan-wehi]

pip install --user --upgrade requests fixed this for me too.

[kapilkd13]

@tetron @mr-c Current version of requests is 2.18.3. We can set min requests version to 2.17.0 and close this issue. What do you suggest.

[tetron]

Debian 9 ships with requests==2.12.4. How about that?

[kapilkd13]

Sounds good. I will do that.