[Flake] TestEventsRPC is flaky

[tejal29]

I have seen this multiple times

rpc_test.go:93: error retrieving event log: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp :36288: connect: connection refused"
    helper.go:194: Skaffold log:
         Listing files to watch...
        >  - test-dev
        > Generating tags...
        >  - test-dev -> gcr.io/k8s-skaffold/test-dev:v1.29.0-35-gba4e2a8-dirty
        > Starting build...
        > Found [kind-kind] context, using local docker daemon.
        > Building [test-dev]...
        > Sending build context to Docker daemon  3.072kB
        > Step 1/3 : FROM busybox
        >  ---> 69593048aa3a
        > Step 2/3 : COPY foo /foo
        >  ---> Using cache
        >  ---> 608862fe6e54
        > Step 3/3 : CMD while true; do cat /foo; sleep 1; done
        >  ---> Using cache
        >  ---> c302772802f6
        > Successfully built c302772802f6
        > Successfully tagged gcr.io/k8s-skaffold/test-dev:v1.29.0-35-gba4e2a8-dirty
        > Starting test...
        > Tags used in deployment:
        >  - test-dev -> gcr.io/k8s-skaffold/test-dev:c302772802f6c4ce0c7d4d200f03fdcd0cdac4ac74ae980763254fdd55ebb205
        > Starting deploy...
        > Loading images into kind cluster nodes...
        >  - gcr.io/k8s-skaffold/test-dev:c302772802f6c4ce0c7d4d200f03fdcd0cdac4ac74ae980763254fdd55ebb205 -> Found
        > Images loaded in 194.78357ms
        >  - deployment.apps/test-dev created
        > Press Ctrl+C to exit
        > Watching for changes...
        > [test-dev]foo
        > [test-dev]foo
        > [test-dev]foo
        > [test-dev]foo
        > [test-dev]foo
        > [test-dev]foo
        > [test-dev]foo
        > [test-dev]foo
        > [test-dev]foo
        > [test-dev]foo
        > [test-dev]foo
        > [test-dev]foo
        > [test-dev]foo
        > [test-dev]foo
        > [test-dev]foo
        > [test-dev]foo
        > 
--- FAIL: TestEventsRPC (20.84s)

[ahmetb]

I've reproed this locally with -count=100.

The issue seems to be what I mentioned the other day in the meeting: When you specify a PORT and it is occupied, skaffold chooses to pick another PORT.

So the output just before what you pasted above is likely something like:

time="2021-08-12T13:51:28-07:00" level=info msg="Running [skaffold dev --namespace skaffoldrshwc --default-repo gcr.io/k8s-skaffold --cache-artifacts=false --rpc-port 59013 --status-check=false] in testdata/dev"
time="2021-08-12T13:51:28-07:00" level=warning msg="starting gRPC server on port 59014. (59013 is already in use)"

as a result it will loop to connect and it will fail:

    rpc_test.go:87: waiting for connection...
    rpc_test.go:87: waiting for connection...
    rpc_test.go:93: error retrieving event log: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp :59013: connect: connection refused"

Do we have any strong feelings around keeping this "skaffold picks a different available port" logic? IMO it makes the tool less predictable and there's no way to communicate the picked port to the caller (short of caller parsing the unstructured output = undesirable?). @nkubala @tejal29

[ahmetb]

/assign

[ahmetb]

Same root cause as #6375.