Unable to process parallel requests over the same session

[mirceaulinic]

When issuing two concomitant requests over the same eAPI session I can get the following errors:

  File "/usr/local/lib/python2.7/dist-packages/pyeapi/client.py", line 730, in run_commands
    response = self._connection.execute(commands, encoding, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 495, in execute
    response = self.send(request)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 388, in send
    response = self.transport.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1099, in getresponse
    raise ResponseNotReady()
ResponseNotReady

  File "/usr/local/lib/python2.7/dist-packages/pyeapi/client.py", line 730, in run_commands
    response = self._connection.execute(commands, encoding, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 495, in execute
    response = self.send(request)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 372, in send
    self.transport.putrequest('POST', '/command-api')
  File "/usr/lib/python2.7/httplib.py", line 921, in putrequest
    raise CannotSendRequest()
CannotSendRequest

or

  File "/usr/local/lib/python2.7/dist-packages/pyeapi/client.py", line 730, in run_commands
    response = self._connection.execute(commands, encoding, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 495, in execute
    response = self.send(request)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 392, in send
    response_content = response.read()
  File "/usr/lib/python2.7/httplib.py", line 578, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 636, in _read_chunked
    value.append(self._safe_read(chunk_left))
  File "/usr/lib/python2.7/httplib.py", line 693, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
AttributeError: 'NoneType' object has no attribute 'read'

For the moment I would only like to understand if this is known, desired or should be corrected.

Thanks!

[mirceaulinic]

Just to provide an example with logs. Starting two parallel jobs using the same pyeAPI session: one that applies a configuration change by creating a new configuration session, the other one invokes the show arp command:

2017-08-29 16:54:48,324 [pyeapi.eapilib   ][DEBUG   ][23270] Request content: {"params": {"format": "json", "version": 1, "cmds": ["enable", "show configuration sessions"]}, "jsonrpc": "2.0", "method": "runCmds", "id": "140371987725072"}
2017-08-29 16:54:49,063 [pyeapi.eapilib   ][DEBUG   ][23270] Response: status:200, reason:OK
2017-08-29 16:54:49,063 [pyeapi.eapilib   ][DEBUG   ][23270] Response content: {"jsonrpc": "2.0", "result": [{}, {"maxSavedSessions": 1, "maxOpenSessions": 5, "sessions": {}}], "id": "140371987725072"}
2017-08-29 16:54:49,064 [pyeapi.eapilib   ][DEBUG   ][23270] eapi_response: {u'jsonrpc': u'2.0', u'result': [{}, {u'maxSavedSessions': 1, u'maxOpenSessions': 5, u'sessions': {}}], u'id': u'140371987725072'}
2017-08-29 16:54:49,064 [pyeapi.eapilib   ][DEBUG   ][23270] Request content: {"params": {"format": "json", "version": 1, "cmds": ["enable", "configure session napalm_324721", "ntp server 1.2.3.4"]}, "jsonrpc": "2.0", "method": "runCmds", "id": "140371987725072"}
2017-08-29 16:54:49,704 [pyeapi.eapilib   ][DEBUG   ][23270] Request content: {"params": {"format": "json", "version": 1, "cmds": ["enable", "show arp"]}, "jsonrpc": "2.0", "method": "runCmds", "id": "140371987725072"}
2017-08-29 16:54:49,704 [/usr/lib/python2.7/dist-packages/salt/utils/napalm.pyc][ERROR   ][23270] Cannot execute "get_arp_table" on edge01.bjm01 as username. Reason: !
2017-08-29 16:54:49,704 [/usr/lib/python2.7/dist-packages/salt/utils/napalm.pyc][ERROR   ][23270] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/napalm.py", line 155, in call
    out = getattr(napalm_device.get('DRIVER'), method)(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/napalm_eos/eos.py", line 801, in get_arp_table
    ipv4_neighbors = self.device.run_commands(commands)[0].get('ipV4Neighbors', [])
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/client.py", line 730, in run_commands
    response = self._connection.execute(commands, encoding, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 495, in execute
    response = self.send(request)
  File "/usr/local/lib/python2.7/dist-packages/pyeapi/eapilib.py", line 372, in send
    self.transport.putrequest('POST', '/command-api')
  File "/usr/lib/python2.7/httplib.py", line 921, in putrequest
    raise CannotSendRequest()
CannotSendRequest

2017-08-29 16:54:50,195 [pyeapi.eapilib   ][DEBUG   ][23270] Response: status:200, reason:OK
2017-08-29 16:54:50,196 [pyeapi.eapilib   ][DEBUG   ][23270] Response content: {"jsonrpc": "2.0", "result": [{}, {}, {}], "id": "140371987725072"}
2017-08-29 16:54:50,196 [pyeapi.eapilib   ][DEBUG   ][23270] eapi_response: {u'jsonrpc': u'2.0', u'result': [{}, {}, {}], u'id': u'140371987725072'}
2017-08-29 16:54:50,196 [pyeapi.eapilib   ][DEBUG   ][23270] Request content: {"params": {"format": "text", "version": 1, "cmds": ["enable", "show session-config named napalm_324721 diffs"]}, "jsonrpc": "2.0", "method": "runCmds", "id": "140371987725072"}
2017-08-29 16:55:03,757 [pyeapi.eapilib   ][DEBUG   ][23270] Response: status:200, reason:OK

The first one continues as normal, while the other one raises CannotSendRequest.

[mharista]

Hi @mirceaulinic,

Is this being done with parallel threads or processes that are both using the same session and simultaneously sending commands? I believe the requests package used for the session in pyeapi is blocking until a requests entire response has been downloaded.

More info here

[mirceaulinic]

Hi @mharista - thanks for looking into this issue.
Yes, this is from parallel processes (same behaviour with multiple threads) sharing the same session.
I went through the resource you pointed out, but I couldn't find anything similar, or describing such a limitation. Also, the errors I am facing, don't suggest they are blocking till the previous response has been downloaded, but simply dropping the a request that arrives the latter. Is this intended by design?

[mirceaulinic]

I am also a bit confused you mentioned the requests library although the traceback doesn't show like pyeapi uses it (requirements.txt confirms that as well). Can you please clarify?

[mharista]

@mirceaulinic My mistake. I assumed pyeapi was using the requests module because of other libraries we maintain that do use it, but pyeapi actually uses httplib. Sorry for causing unnecessary confusion. The error you are hitting is similar to what I mentioned previously. The second thread is likely making another request on the session before the original thread has made a call to getresponse(). I believe this is intended by design in python's httplib module and inherited by pyeapi.

[mirceaulinic]

Thanks for confirming @mharista.

AFAIK using requests we shouldn't see anything like that, but I need to double check. Before starting to work on anything like that, would you welcome a PR to improve this side and switch from httplib to requests?

[mharista]

@mirceaulinic from the small amount I've looked into this, the library recommended for asynchronous requests is grequests. The link I provided previously mentions this.

Blocking Or Non-Blocking?
With the default Transport Adapter in place, Requests does not provide any kind of non-blocking IO. The Response.content property will block until the entire response has been downloaded. If you require more granularity, the streaming features of the library (see Streaming Requests) allow you to retrieve smaller quantities of the response at a time. However, these calls will still block.

If you are concerned about the use of blocking IO, there are lots of projects out there that combine Requests with one of Python's asynchronicity frameworks. Two excellent examples are grequests and requests-futures.

As for a PR to support handling of async requests, I am open to that as long as all the existing functionality is maintained and none of the existing unit/system tests are broken.

[mirceaulinic]

I understand your reasoning and your concerns - makes sense. Just to clarify about the blocking behaviour: at the time being if there are two parallel requests, the second one is not waiting for the first one to finish, but it is dropped immediately. I would rather qualify this as a bug, or improper design -- this is basically in direct contradiction with the HA and concurrency principles. Is this one of the goals?

…

On Fri, Sep 1, 2017, 18:15 mharista ***@***.***> wrote: @mirceaulinic <https://github.com/mirceaulinic> from the small amount I've looked into this, the library recommended for asynchronous requests is grequests. The link I provided previously mentions this. Blocking Or Non-Blocking? With the default Transport Adapter in place, Requests does not provide any kind of non-blocking IO. The Response.content property will block until the entire response has been downloaded. If you require more granularity, the streaming features of the library (see Streaming Requests) allow you to retrieve smaller quantities of the response at a time. However, these calls will still block. If you are concerned about the use of blocking IO, there are lots of projects out there that combine Requests with one of Python's asynchronicity frameworks. Two excellent examples are grequests and requests-futures. As for a PR to support handling of async requests, I am open to that as long as all the existing functionality is maintained and none of the existing unit/system tests are broken. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#150 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AP69lwho7J88-kMHR9drayLQmDvzLpgHks5seDu2gaJpZM4PGJyD> .

[mharista]

@mirceaulinic is the question relating to the requests module, httplib or pyeapi? Pyeapi uses httplib which by design doesn't support async requests at the moment.

[mirceaulinic]

Hi @mharista - my question is related to pyeapi: choosing httplib in favour of other solutions is a design choice.

[mharista]

@mirceaulinic Could you please contact me via email? mhartzel@arista.com

[mharista]

Hi @mirceaulinic, Is this still an issue you would like to discuss?