[PR] Switch to aiohttp and full asynchronous I/O in the core

[kopf-archiver]

A pull request by nolar at 2019-11-13 10:31:58+00:00
Original URL: zalando-incubator/kopf#227
Merged by nolar at 2019-11-13 11:00:51+00:00

Remove any synchronous Kubernetes API clients or asyncio executors. Make Kopf fully and truly asynchronous.

Issue : #142, #140, maybe #204, maybe #169
Replaces: #176 #152 #143, #141

Problem

pykube-ng, kubernetes, requests, and any other synchronous client libraries use the streaming responses of the built-in urllib3 and http for watching over the k8s-events.

These streaming requests/responses can be closed when a chunk/line is yielded to the consumer, the control flow is returned to the caller, and the streaming socket itself is idling. E.g., for requests: https://2.python-requests.org/en/master/user/advanced/#streaming-requests

However, if nothing happens on the k8s-event stream (i.e. no k8s resources are added/modified/deleted), the streaming response spends most of its time in the blocking read() operation on a socket. It can remain there for long time — minutes, hours — until some data arrives on the socket.

If the streaming response runs in a thread, while the main thread is used for an asyncio event-loop, such stream cannot be closed/cancelled/terminated (because of the blocking read()). This, in turn, makes the application to hang on exit, holding its pod from restarting, since the thread is not finished until the read() call is finished.

There is no easy way to terminate the blocking read() operation on a socket. One way is a dirty hack with the OS-level process signals, which interrupt the I/O operations on low level (OS&libc&co) — see #152.

Solution

The proper solution, however, is to use async i/o inside of the async app.

This PR converts all i/o to/from Kubernetes API to aiohttp. It is already present in the dependencies indirectly.

This efficiently removes pykube-ng (or any other clients) from the Kopf's core. There is no much need in them, as the main purpose of the client libraries is to provide a convenient DSL (domain-specific language) for the Kubernetes objects manipulation. In Kopf, all manipulation is unified, used only internally, not exposed as a public interface, and implemented so that the kind of objects being handled is not so important.

An attempt to do this was already performed in #176, but it contained a huge part about authentication methods. With custom authentication and piggybacking implemented separately in #226. This new PR now contains only the i/o-related changes.

Testing

For testing, aresponses is used instead of mocked requests. It runs an actual web server locally, and intercepts all aiohttp outgoing requests to be redirected to that web-server.

This switch led to almost full rewrite of all tests for kopf.clients module (all API communication) — which makes a half of this PR's size (while keeping the same semantics of the tests).

Types of Changes

• Bug fix (non-breaking change which fixes an issue)
• Refactor/improvements

Review

List of tasks the reviewer must do to review the PR

• Tests
• Documentation