Flaky test: Test/GracefulClientOnGoAway

[menghanl]

Two recently merged suspicious changes:

• stream: fix calloption.After() race in finish #3672 stream: fix calloption.After() race in finish
• retry: prevent per-RPC creds error from being transparently retried #3677 retry: prevent per-RPC creds error from being transparently retried

Log

    --- FAIL: Test/GracefulClientOnGoAway (0.32s)
        tlogger.go:97: clientconn.go:162: Channel Created
        tlogger.go:97: clientconn.go:248: parsed scheme: ""
        tlogger.go:97: clientconn.go:254: scheme "" not registered, fallback to default scheme
        tlogger.go:97: resolver_conn_wrapper.go:143: ccResolverWrapper: sending update to cc: {[{127.0.0.1:43503  <nil> 0 <nil>}] <nil> <nil>}
        tlogger.go:97: resolver_conn_wrapper.go:218: Resolver state updated: {Addresses:[{Addr:127.0.0.1:43503 ServerName: Attributes:<nil> Type:0 Metadata:<nil>}] ServiceConfig:<nil> Attributes:<nil>} (resolver returned new addresses)
        tlogger.go:97: clientconn.go:678: ClientConn switching balancer to "pick_first"
        tlogger.go:97: clientconn.go:693: Channel switches to new LB policy "pick_first"
        tlogger.go:97: clientconn.go:734: Subchannel Created
        tlogger.go:97: clientconn.go:734: Subchannel(id:1082) created
        tlogger.go:97: clientconn.go:1067: Subchannel Connectivity change to CONNECTING
        tlogger.go:111: picker_wrapper.go:162: blockingPicker: the picked transport is not ready, loop back to repick
        tlogger.go:97: clientconn.go:1204: Subchannel picks a new address "127.0.0.1:43503" to connect
        tlogger.go:97: clientconn.go:425: Channel Connectivity change to CONNECTING
        tlogger.go:97: clientconn.go:1067: Subchannel Connectivity change to READY
        tlogger.go:97: clientconn.go:425: Channel Connectivity change to READY
        tlogger.go:97: clientconn.go:1067: Subchannel Connectivity change to CONNECTING
        tlogger.go:97: clientconn.go:1204: Subchannel picks a new address "127.0.0.1:43503" to connect
        tlogger.go:111: picker_wrapper.go:162: blockingPicker: the picked transport is not ready, loop back to repick
        tlogger.go:97: clientconn.go:425: Channel Connectivity change to CONNECTING
        tlogger.go:97: clientconn.go:1067: Subchannel Connectivity change to READY
        tlogger.go:97: clientconn.go:425: Channel Connectivity change to READY
        tlogger.go:97: clientconn.go:1067: Subchannel Connectivity change to CONNECTING
        tlogger.go:97: clientconn.go:1204: Subchannel picks a new address "127.0.0.1:43503" to connect
        tlogger.go:97: clientconn.go:425: Channel Connectivity change to CONNECTING
        tlogger.go:97: clientconn.go:1067: Subchannel Connectivity change to READY
        tlogger.go:97: clientconn.go:425: Channel Connectivity change to READY
        tlogger.go:97: clientconn.go:1067: Subchannel Connectivity change to CONNECTING
        tlogger.go:97: clientconn.go:1204: Subchannel picks a new address "127.0.0.1:43503" to connect
        tlogger.go:97: clientconn.go:425: Channel Connectivity change to CONNECTING
        tlogger.go:97: clientconn.go:1067: Subchannel Connectivity change to READY
        tlogger.go:97: clientconn.go:425: Channel Connectivity change to READY
        goaway_test.go:70: EmptyCall(_, _) = _, rpc error: code = Unavailable desc = transport is closing; want _, <nil>
        tlogger.go:97: clientconn.go:425: Channel Connectivity change to SHUTDOWN
        tlogger.go:97: clientconn.go:1067: Subchannel Connectivity change to SHUTDOWN
        tlogger.go:97: clientconn.go:1431: Subchannel Deleted
        tlogger.go:97: clientconn.go:1431: Subchanel(id:1082) deleted
        tlogger.go:97: clientconn.go:1023: Channel Deleted
        tlogger.go:111: types.go:69: attempt to delete child with id 1082 from a parent (id=1081) that doesn't currently exist
        tlogger.go:111: types.go:69: attempt to delete child with id 1082 from a parent (id=1081) that doesn't currently exist

[dfawley]

If I understood everything correctly, #3677 should be safe, since there are no per-RPC creds in this test, and that was the only way NewStream could fail (and consequently make attempt.s be nil. (We should still confirm this with testing anyway, in case there is a path I didn't see or understand.)

[dfawley]

This flaked again in a cron run:

https://github.com/grpc/grpc-go/runs/2660658084#step:8:160

[easwars]

https://github.com/grpc/grpc-go/actions/runs/3953486390/jobs/6769812156

[zasweq]

https://github.com/grpc/grpc-go/actions/runs/4473574106/jobs/7861110092?pr=6141