socket.gaierror on requests even though connector resolve=False

[digitaldavenyc]

So this is my setup for retrieving a page

connector = aiohttp.TCPConnector(verify_ssl=False, share_cookies=False, resolve=False)
r = aiohttp.request(method='GET', url=data['url'], connector=connector)
result = yield from asyncio.wait_for(r, timeout=10)

This is error is periodically thrown

future: Future<exception=gaierror(-2, 'Name or service not known')>
Traceback (most recent call last):
  File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.4/socket.py", line 530, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

I'm not sure why getaddrinfo is being called because I have resolve host marked as false. I'd prefer it if this was not called at all since I do not need this information but if there is a way to wrap it in a try exception that would be helpful as well.

[asvetlov]

@digitaldavenyc resolve parameter has opposite meaning.
If it's True TCPConnector does cache DNS queries.
Perhaps we need rename it to cache_dns or something like this.
But you still need first DNS lookup for unknown hosts, isn't it?

[digitaldavenyc]

@asvetlov DNS information isn't always necessary. Just to do a GET request, I don't need DNS information.

I think renaming it to cache_dns would be a bit more explicit, I assumed marking resolve False wouldn't make the DNS request.

The real issue here is how to catch the error thrown, I'm of the opinion aiohttp library needs to catch the socket.gaierror error and raise an exception. Wrapping the aiohttp.request in a try except doesn't work since it's a generator.

[asvetlov]

socket.gaierror is subclass of OSError starting from Python 3.3.
Sometimes aiohttp may raise OSError, not own internal exception. I think it's fine.

Yes, aiohttp.request is generator (BTW I recommend considering aiohttp.ClientSession instead of request).
You may catch errors from request by

try:
   resp = yield from aiohttp.request(...)
except OSError as exc:
   process_exception(exc)

[asvetlov]

I'm working on patch for cache_dns and family.

[digitaldavenyc]

I'll give it a shot but I'm not sure if wrapping the aiohttp.request(...) in a try except is going to work. Here is a more details of the code...

try:
   connector = aiohttp.TCPConnector(verify_ssl=False, share_cookies=False, resolve=False)
   r = aiohttp.request(method='GET', url=data['url'], connector=connector)
   result = yield from asyncio.wait_for(r, timeout=10)
except (TimeoutError, aiohttp.errors.ClientResponseError, socket.gaierror) as e:
   print(e.__class__.__name__)

The socket.gaierror is still thrown and crashes the application. If you look at the error in my original post, it doesn't mention the aiohttp library either.

[asvetlov]

Did you use IP addresses or DNS names in your code?
I mean 8.8.8.8 or google-public-dns-a.google.com.
For last case throwing gaierror doesn't surprise me.

[digitaldavenyc]

I'm making requests with full http URLs, not DNS names or IP addresses but I assume the library parses the URL and uses the hostname to get the DNS address. Is there anyway to turn this feature off?

[asvetlov]

If you URL has DNS name as HOST the system (aiohttp+asyncio+python) should pass it through DNS resolver. The last unsurprisingly may raise gaierror.

Or, maybe, I don't understand you clean.
Would you present code snippet which I run on my laptop to reproduce your problem?

[asvetlov]

See also #415 for new names

[asvetlov]

Fixed by #415

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.