Network in containers breaks under bigger network load

[mjkonarski-b]

Network breaks in containers when they start multiple network connections at the same time.

I noticed this behaviour e.g. during downloading Python dependencies. When multiple packages are downloaded at the same time I start getting Network is unreachable error. Then when I login to the underlying QEMU machine (limactl shell colima) I can see that it can't reach any network address. I cannot even ping 8.8.8.8. My host computer doesn't have any connection issues.

It gets better after few moments of inactivity. Restarting QEMU machine (colima stop && colima start) fixes the network, but the problem comes back when I increase the network load.

This is a problem that I can consistently reproduce. I created a minimum setup to demonstrate it: https://github.com/mjkonarski-b/colima-poc

I experience that problem on multiple Macbooks, so it doesn't seem to be related to any particular processor or macOS version:

• MBP 2021 M1 Pro with 12.1 Monterey
• MBP 2019 i7 with 12.1 Monterey
• MBP 2019 i7 with 11.5.2 BigSur

$ colima version
colima version 0.3.2
git commit: 272db4732b90390232ed9bdba955877f46a50552

runtime: docker
arch: aarch64
client: v20.10.11
server: v20.10.11


$ limactl --version
limactl version 0.8.1


$ qemu-img --version
qemu-img version 6.2.0
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

[dmikusa]

+1 I'm seeing the same thing. Same colima, lima and qemu versions. Mac 12.1 Monterey

1. colima start
2. docker pull any image that's a couple of hundred MB+
3. It gets halfway through and stalls. It'll try to continue with limited success.

I tried downgrading to colima 0.3.1 and it didn't seem to help. I did a colima delete and colima start after downgrading but still seeing the issue.

[mjkonarski-b]

This seems to be related too #137

[mjkonarski-b]

I did more investigation, but I couldn't find the root cause. So far it seems that the problem lies in Lima or QEMU itself. I could reproduce it on machines running raw Lima images, without Colima. I found two issues in Lima repo that seem describing the very same problem:
lima-vm/lima#537
lima-vm/lima#561

[bolt-juri-gavshin]

I believe the problem is not inside Docker or containers, maybe even not in Co(lima)/qemu...

Some additional information:
I have Colima version 0.3.2.
Everything worked in MacOS 12.0.1, but broke after upgrade to 12.2.

Steps to reproduce:

colima delete
colima start --cpu 6 --memory 8 --disk 60
colima ssh -- sudo apk add apache2-utils
colima ssh -- ab -c 100 -n 1000 https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl

I receive lots of errors like SSL read failed (5) - closing connection and the summary is:

Time taken for tests:   0.916 seconds
Complete requests:      1000
Failed requests:        1818
   (Connect: 0, Receive: 0, Length: 909, Exceptions: 909)

If last command is run on the host, everything is good:

Time taken for tests:   10.788 seconds
Complete requests:      1000
Failed requests:        0

Before MacOS upgrade (12.0.1 -> 12.2) running ab inside Colima VM gave the same result, as host (i.e. success).

P.S. In Docker problem is the same, for example:

docker run --rm jordi/ab -c 100 -n 1000 https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl

[bolt-juri-gavshin]

Another note: Rancher Desktop 1.0.0 works without problems on the same machine, when I run docker command:

docker run --rm jordi/ab -c 100 -n 1000 https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl

[mrsladek]

Hi,
Same issue here on 5 different MBP machines.

When pulling multiple images at the same time with docker-compose the network breaks and I get unreachable error or i/o timeout.

Great if the problem could be addressed soon.

$ colima version 0.3.2
git commit: 272db4732b90390232ed9bdba955877f46a50552

runtime: docker
arch: x86_64
client: v20.10.12
server: v20.10.11

$ limactl --version                                                                                                                                                                                          
limactl version 0.8.1

$ qemu-img --version                                                                                                                                                                                         
qemu-img version 6.2.0
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

[brunoselvacj]

That looks like a problem with Alpine itself. I did the same kind of configuration that lima does using yaml file straight to limavm and the results are the same. That is not reproducible with Ubuntu using the yaml and loading docker as provisioning

$ colima version 0.3.2
git commit: 272db47

runtime: docker
arch: x86_64
client: v20.10.12
server: v20.10.11

$ limactl --version
limactl version 0.8.1

$ qemu-img --version
qemu-img version 6.2.0
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

[MMartyn]

I was having this issue and was able to work around it by adding the following to ~/.lima/_config/override.yaml

useHostResolver: false
dns:
- 8.8.8.8

[abiosoft]

I was having this issue and was able to work around it by adding the following to ~/.lima/_config/override.yaml

useHostResolver: false
dns:
- 8.8.8.8

If that's the case, user-configurable dns can be added to the next version.

[mrsladek]

useHostResolver: false
dns:

• 8.8.8.8

not working on my setup :(

[abiosoft]

useHostResolver: false
dns:

• 8.8.8.8

not working on my setup :(

I'm working on a proper fix for configurable dns on each startup.
This actually takes effect on the initial VM creation, so nothing changes if you apply this to an existing VM.

[bolt-juri-gavshin]

I'm working on a proper fix for configurable dns on each startup.

The problem doesn't seem to be only DNS related. As author described, after VM gets into a "bad" state, ping/connection by IP doesn't work either.

As I wrote before, rancher desktop (which also uses alpine lima images under the hood) works fine on the same machine. Maybe we can try to use the same images? I am ready to experiment with new versions, just need some guidance. I have M1, M1Pro and Intel Macs at my disposal to test it.

[keerati]

I was having this issue and was able to work around it by adding the following to ~/.lima/_config/override.yaml

useHostResolver: false
dns:
- 8.8.8.8

This works for me.

[dmikusa]

I was having this issue and was able to work around it by adding the following to ~/.lima/_config/override.yaml

useHostResolver: false
dns:
- 8.8.8.8

I got some time to try this with my previous docker pull test and it seems better, usable even, but I did see a couple of times where docker had to retry, which never happens when I run in other VMs.

[abiosoft]

Another note: Rancher Desktop 1.0.0 works without problems on the same machine, when I run docker command:

docker run --rm jordi/ab -c 100 -n 1000 https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl

I actually got a similar experience with Rancher Desktop. It seems to be something specific to Alpine as I could not reproduce with an Ubuntu image.

I am still troubleshooting and would prefer not to ditch Alpine.