Make client more async-friendly

[whereswaldon]

See #26 for discussion.

I'll be honest; this is over my head right now. I've never done any Python async stuff. I'd love for anyone with more experience to chime in with thoughts or ideas about how/why to do this.

[ntninja]

Well this seems to be split in two parts actually:

• Greenthread-style async
• EventLoop-style async

Since we are using Python requests (for better or for worse), adding the Greenthread-style async (using gevent) seems to be fairly simple: We just have to first try to use grequests (which patches up classic requests to use gevent for blocking I/O) in ipfsApi/http.py, then fall back to classic requests if that is not available. Users that need async then can simply depend on grequests to have greenthreading enabled for this library.

EventLoop-style async (the thing with callbacks / futures), is a lot harder as it (1) requires actual code change on our side, (2) requires quite some code change inside of requests, (3) requires a main loop (which only part of Python 3.4+ by default). I don't think this is feasible (without lots of uglyness), unless we want to make the library go async-only internally (a blocking interface can still be easily exposed on top of that) and replace requests as a whole while we're at it.

Why to do this? Paralysing requests, using less (one) threads, … Async is just generally more efficient, if you're doing more than one thing at once (that's why people do it in general). For the "script automation"-style case it doesn't matter, but for more "serious" applications it tends to give you much better integration (no need to stuff everything into one big thread callback) and nicer code.

[jbenet]

I strongly recommend green threads (gevent or eventlet) over eventloops.

[ntninja]

@jbenet: Due to their simpler integration, or because of some other experience/reason?

[jbenet]

Yes, events are a major engineering time waster. Even the most ardent
proponents couldn't beat themselves programming with user level
threads/coroutines. (This is anecdotal evidence I collected by asking
several of them.)

Evented code is great for simple FSMs and simple network protocols, but
quickly gets complex as the caller. Hence user level threads/coroutines +
callback handlers where necessary is a happy medium.
On Mon, Aug 15, 2016 at 18:32 Alexander Schlarb notifications@github.com
wrote:

@jbenet https://github.com/jbenet: Due to their simpler integration, or
because of some other experience/reason?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
#59 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAIcobL9pQzw1JarfeR1crv04J-fO9jPks5qgOjugaJpZM4Jky7-
.

[ntninja]

@jbenet: The more I read your second answer, the more confused I get. Correct me if I'm wrong: Your basically saying that we should support greenthreads & coroutines and not just callbacks?
To clarify: My first suggestion was about only integrating with greenthread libraries (that are not part of standard python), while my second suggestion was about supporting Python's native event loop adaptor (that can run on top of pretty much any existing event loop OR its own) and coroutine support. Which, by extension, would allow developers using greenthreads to also leverage the library using a bit of glue code (see aioeventlet for an example). I never talked about using some custom I/O event loop such as the (deprecated) asyncore module.

[jbenet]

@Alexander255 I was mostly talking about events vs (kernel)threads vs {user,green}threads. all that you say sounds good. i haven't used the new coroutine asyncio lib in py3, but image it works well. (though the explicit await is usually a sign of "coroutines added too little too late" to a language)

what i would suggest is: make it easy on programmers. whatever is the easiest to debug will save the most time. people will optimize other parts as they need to, but the most important thing is making sure things work well and robustly for people. (so whatever is most idiomatic and usable to people. that may be either your prop 1 or prop 2 depending on what people like to do in python these days)

[ntninja]

Yes, the explicit await is somewhat redundant, but without it you couldn't do things like asyncio.gather or storing a coroutine for later. But your also right that Python was designed for single-core blocking I/O in the 90s and that that legacy definitely is showing.

Based on your "make it easy" suggestion, I'd go with 2 (greenthreading has also been somewhat nichey in Python and with ayncio gaining a lot of traction in the last years, I can't image that to change now). However, since there's no asyncio for Python 2 (and in particular nothing similar to aiohttp), it will need a multiple backend approach unfortunately. 😢
We will not be able to drop Python 2.7 support for several years unfortunately, but I'll not go out of my way to add support for that legacy platform.

[ntninja]

@faisalburhanudin: I have just stumbled upon this great-looking thing by accident and got interested: What was your inspiration in writing this? Why did you (apparently) abandon it? It looks like an important milestone for the Async interface we still never yet had, so it may be useful to us as well.

Regarding usefulness in particular: Would you be willing to license this code under MIT (or similar), in case we can reuse any code for our own asyncio port?

Good job on that library either way! 😁

[ntninja]

I should mention that there is, and has been for a while, a branch of mine with code to implement this.