oc port-forward concurrency limitation

[sosiouxme]

When attempting multiple connections over a port forwarded to a container (oc port-forward), there seems to be a limit of 5 on the number of concurrent connections. Once there are more, connections hang / timeout and don't seem to recover.

This was encountered with a nodejs container in trying to use the node-inspector debugger app which listens on two non-public ports, the HTTP port presenting an HTML5 webapp that loads a number of assets concurrently. Here is typical output from the port-forward:

I0819 13:28:57.321178   18067 portforward.go:225] Forwarding from 127.0.0.1:8080 -> 8080
I0819 13:28:57.321270   18067 portforward.go:225] Forwarding from [::1]:8080 -> 8080
I0819 13:28:57.321298   18067 portforward.go:225] Forwarding from 127.0.0.1:5858 -> 5858
I0819 13:28:57.321327   18067 portforward.go:225] Forwarding from [::1]:5858 -> 5858
I0819 13:29:13.399831   18067 portforward.go:251] Handling connection for 8080
I0819 13:29:37.633514   18067 portforward.go:251] Handling connection for 8080
I0819 13:29:37.633844   18067 portforward.go:251] Handling connection for 8080
I0819 13:29:37.633880   18067 portforward.go:251] Handling connection for 8080
I0819 13:29:37.633893   18067 portforward.go:251] Handling connection for 8080
I0819 13:29:38.107638   18067 portforward.go:251] Handling connection for 8080
I0819 13:29:38.108131   18067 portforward.go:251] Handling connection for 8080
E0819 13:30:07.719444   18067 portforward.go:310] An error occurred forwarding 8080 -> 8080: Timed out waiting for data stream
E0819 13:30:07.795932   18067 portforward.go:310] An error occurred forwarding 8080 -> 8080: Timed out waiting for data stream
E0819 13:30:07.795949   18067 portforward.go:310] An error occurred forwarding 8080 -> 8080: Timed out waiting for data stream
I0819 13:30:07.796389   18067 portforward.go:251] Handling connection for 8080
I0819 13:30:07.796474   18067 portforward.go:251] Handling connection for 8080
E0819 13:30:08.201089   18067 portforward.go:280] Error creating forwarding stream for port 8080 -> 8080: Timeout occured
E0819 13:30:08.201713   18067 portforward.go:280] Error creating forwarding stream for port 8080 -> 8080: Timeout occured
E0819 13:30:37.796628   18067 portforward.go:262] Error creating error stream for port 8080 -> 8080: Timeout occured

Individual serialized requests succeed indefinitely. Only concurrency seems to be an issue. This makes the debugger entirely unusable. Since it assumes two ports on the same host I can't even expose it usefully via service/route as a workaround. I haven't tried it with any other application yet; it's possible it would work differently.

[jcantrill]

cc @maxandersen FYI

[maxandersen]

ouch ;/

[liggitt]

@ncdc is this caused by the spdystream FRAME_WORKERS being set to 5 or is that just a coincidence?

[ncdc]

I haven't had any time to investigate yet. I sure hope not :-/

On Wed, Sep 2, 2015 at 12:02 PM, Jordan Liggitt notifications@github.com
wrote:

@ncdc https://github.com/ncdc is this caused by the spdystream
FRAME_WORKERS being set to 5 or is that just a coincidence?

—
Reply to this email directly or view it on GitHub
#4287 (comment).

[ncdc]

It doesn't seem to be a limit of 5. I think there may be some sort of partial deadlock occurring. If I run 10 or even 100 simultaneous curls against nginx, several complete successfully, but after a certain point, attempts to handle new forwarded connections hang with various timeouts occurring. Each connection should result in 2 streams being created: 1 for the server to convey errors back to the client, and the other to carry the actual forwarded data. When things start hanging, stream creation times out, and it waits up to 30 seconds for the server to reply that it received a new stream. I'll keep investigating.

[ncdc]

I have found the root cause and have a fix locally that seems to work. I need to clean it up a bit and I'll have something to submit upstream next week if all goes well.

[ncdc]

Upstream issue & analysis: kubernetes/kubernetes#13673

1 important item to note - my proposed fix will make it so older clients won't be able to do port forwarding against newer servers containing the fix. I'm wondering how much of an issue this will be?

[liggitt]

I really don't like the client-side allocation of stream ids... any way around that?

edit: I came around. Finishing up discussion in kubernetes/kubernetes#13673

[ncdc]

kubernetes/kubernetes#12283 just merged upstream.

@brenton @danmcp @sdodson @smarterclayton @sosiouxme @bparees @pweil- @liggitt do you want this cherry picked to origin now, or will we be doing another rebase that could pick this up?