[WCOW] Support live restore

[Random-Liu]

Ref #1257

This is a must have for containerd windows support I believe.

Today, if you restart containerd when there are containers are running, containerd won't be able to recover control of those containers after restart.

[liyanhui1228]

Thanks for filing this!

Adding notes for work around to here:

crictl pods
crictl stopp POD_ID  # this supports prefix match

# if containerd failed, use ctr instead
ctr -n=k8s.io task ls
ctr -n=k8s.io task kill CONTAINER_ID
ctr -n=k8s.io task rm CONTAINER_ID

[Random-Liu]

I looked into this today, and found that it is a bit complicated.

We are using named pipe for shim logs and container stdio. Today, the problem is that once containerd is restarted, we can't reconnect shim logs and container stdio any more.

Shim Log

Shim is the named pipe server and writer, containerd is the named pipe client and reader. Before microsoft/hcsshim#702, after containerd restarts, containerd tries to dial named pipe, but the shim doesn't accept new connection. After microsoft/hcsshim#702, shim accepts new connection now. However, there are still problems:

• Problem 1: Shim doesn't handle write errors when containerd is down, so those logs will be lost. This problem can be fixed by wrapping the writer to buffer and block when the pipe is broken;
• Problem 2: Logs left in the buffer will be lost, if they were written before containerd is down, but not read out yet.

I don't have a good solution for problem 2 yet. It is possible to call DisconnectNamedPipe and ConnectNamedPipe on the server side to preserve the named pipe instance, however:

• If I read the code correctly, go-winio library doesn't provide the ability to reuse a pipe instance;
• If I read the doc correct, Any unread data in the pipe is discarded after DisconnectNamedPipe is called.

Container Stdio

Shim is the named pipe client and writer for stdout/stderr and reader for stdin, containerd is the named pipe server and reader for stdout/stderr and writer for stdin. After containerd restarts, shim doesn't try to reconnect to containerd. We can easily wrap the writer to block and retry connection when pipe is broken. However, we still have the problem 2 mentioned above.

Then the most important thing is how to handle problem 2, there are several possibilities:

• Find out a way to preserve buffered data and rebuild the named pipe connection. I haven't found a way of doing this by reading Microsoft docs.
• Find out a way to read buffered data out after the pipe is broken. I haven't found a way of doing this by reading Microsoft docs.
• Introduce some ack mechanism so that we know what has been read. This seems complex and error prone.
• Tolerant buffered data loss across containerd restarts. We can try our best to buffer and bolck IO in the shim when containerd is down, but data that is already in pipe buffer will be lost.

@jterry75 I may misunderstood things. Do you have any idea about this?

/cc @yujuhong

[Random-Liu]

On a second thoughts, actually even today, on linux if containerd restarts when the CRI plugin reads logs from FIFO, but hasn't written into log files yet, we may also lose logs.

Maybe it is fine to just admit that buffered logs can be lost when containerd restarts.

But we still need to fix problem 1, because that may lose significant amount of logs if containerd is down for too long time. And in the future, when we move CRI log generation into the shim, we won't have this problem anyway. :) Ref containerd/containerd#3085

[jterry75]

@Random-Liu - So I actually don't think 1 or 2 are problems :). I have already written the code that fixes them and verifies no data loss (up to some amount of buffered logs 1KB for example). We previously had a simple version of that in containerd and it wouldn't be hard to move into our shim. But Containerd Stdio IS the problem IMO. In order for this to work I think we will have to make the shim be the server of stdio as well. It cannot operate as a client in this case. But fundamentally I dont know how to do that because the expectation is so much different than Linux. Basically on Linux its:

Create IO
Create Container
Start

For Windows it will be:
Create Container
Connect IO
Start

For the part about losing data after it has been read by CD but not written out there is nothing we can do :). But its ok because stdio and logs are not state serialized so I dont think anyone has that expectation anyways.

[jterry75]

I have a cleaner version of this code but here is how it worked in containerd before we removed the in-tree shim: containerd/containerd@4b5dfae#diff-cfd0dfd519df5b61374c555f4c404c31L125

[Random-Liu]

Offline discussed with @jterry75, we can use the "blocking buffered writer" he wrote before.

Besides that, one more change we may want to make is to make shim a pipe server for both shim log and container stdio. The reason is that in containerd's model, shim is the long running server, while containerd is the one can restart.

If the server is put into containerd, shim will have to keep retrying reconnect when containerd is down. The busy retry may introduced unnecessary overhead especially that the overhead is per container.

If we make shim the server, it can simply Accept the new connection from restarted containerd, which is much more performant.

[PatrickLang]

I discussed this with @Random-Liu a bit and wanted to add some more background.

I'd like to consider pushing this out - it's not something that's currently implemented with Docker. Today Docker stops containers when it's restarted. ContainerD could be modified to do the same on Windows to prevent leaking Windows containers that it can't manage. This would be a mitigation until live restore could be implemented later.

As of right now I don't think this will be ready and tested for Kubernetes 1.18 / ContainerD 1.3 so I'm proposing moving it out.

[joedborg]

I might be misunderstanding but I think this does work in Docker https://docs.docker.com/config/containers/live-restore/