[HTTP/3] Stress test regression

[CarnaViire]

Last success run was on 7/20
https://dev.azure.com/dnceng/public/_build/results?buildId=1248255&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd&l=700

client_1  | [07/20/2021 14:15:10] Total: 8,238 Runtime: 00:31:00
client_1  | 	 0: GET                       Success: 543	Canceled: 42	Fail: 0
client_1  | 	 1: GET Partial               Success: 539	Canceled: 46	Fail: 0
client_1  | 	 2: GET Headers               Success: 549	Canceled: 37	Fail: 0
client_1  | 	 3: GET Parameters            Success: 532	Canceled: 55	Fail: 0
client_1  | 	 4: GET Aborted               Success: 549	Canceled: 37	Fail: 0
client_1  | 	 5: POST                      Success: 549	Canceled: 37	Fail: 0
client_1  | 	 6: POST Multipart Data       Success: 541	Canceled: 45	Fail: 0
client_1  | 	 7: POST Duplex               Success: 544	Canceled: 42	Fail: 0
client_1  | 	 8: POST Duplex Slow          Success: 544	Canceled: 42	Fail: 0
client_1  | 	 9: POST Duplex Dispose       Success: 547	Canceled: 39	Fail: 0
client_1  | 	10: POST ExpectContinue       Success: 0	Canceled: 43	Fail: 0
client_1  | 	11: HEAD                      Success: 527	Canceled: 57	Fail: 0
client_1  | 	12: PUT                       Success: 545	Canceled: 39	Fail: 0
client_1  | 	13: PUT Slow                  Success: 543	Canceled: 41	Fail: 0
client_1  | 	14: GET Slow                  Success: 536	Canceled: 48	Fail: 0
client_1  | 	    TOTAL                     Success: 7,588	Canceled: 650	Fail: 0

Regression seen from 7/21
https://dev.azure.com/dnceng/public/_build/results?buildId=1250670&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd
The log is big so it's not showing in UI, you'll need 'View raw log' button

It goes like this:

1. First, there are some errors like Stream aborted by peer (4294967295) (means stream was disposed prematurely) or timeout or some other unexpected cancellation. Last statistic I was able to see before all requests will start to fail

2021-07-21T13:59:56.4253843Z client_1  | [07/21/2021 13:59:56] Total: 511,180 Runtime: 00:21:00
2021-07-21T13:59:56.4254938Z client_1  | 	 0: GET                       Success: 33,852	Canceled: 2,482	Fail: 0
2021-07-21T13:59:56.4255872Z client_1  | 	 1: GET Partial               Success: 33,948	Canceled: 2,386	Fail: 0
2021-07-21T13:59:56.4257107Z client_1  | 	 2: GET Headers               Success: 33,835	Canceled: 2,500	Fail: 0
2021-07-21T13:59:56.4258055Z client_1  | 	 3: GET Parameters            Success: 33,925	Canceled: 2,410	Fail: 1
2021-07-21T13:59:56.4260027Z client_1  | 	 4: GET Aborted               Success: 33,816	Canceled: 2,506	Fail: 14
2021-07-21T13:59:56.4260783Z client_1  | 	 5: POST                      Success: 33,840	Canceled: 2,494	Fail: 2
2021-07-21T13:59:56.4261487Z client_1  | 	 6: POST Multipart Data       Success: 33,839	Canceled: 2,496	Fail: 1
2021-07-21T13:59:56.4262369Z client_1  | 	 7: POST Duplex               Success: 33,914	Canceled: 2,422	Fail: 0
2021-07-21T13:59:56.4263120Z client_1  | 	 8: POST Duplex Slow          Success: 33,880	Canceled: 2,454	Fail: 2
2021-07-21T13:59:56.4263939Z client_1  | 	 9: POST Duplex Dispose       Success: 33,776	Canceled: 2,560	Fail: 0
2021-07-21T13:59:56.4264675Z client_1  | 	10: POST ExpectContinue       Success: 0	Canceled: 2,491	Fail: 0
2021-07-21T13:59:56.4265339Z client_1  | 	11: HEAD                      Success: 33,889	Canceled: 2,443	Fail: 2
2021-07-21T13:59:56.4265994Z client_1  | 	12: PUT                       Success: 33,855	Canceled: 2,477	Fail: 2
2021-07-21T13:59:56.4266650Z client_1  | 	13: PUT Slow                  Success: 33,641	Canceled: 2,691	Fail: 1
2021-07-21T13:59:56.4267299Z client_1  | 	14: GET Slow                  Success: 33,980	Canceled: 2,352	Fail: 1
2021-07-21T13:59:56.4267950Z client_1  | 	    TOTAL                     Success: 473,990	Canceled: 37,164	Fail: 26

2. Then, come errors of aborting because of premature connection close. After that, as mentioned, no new requests would succeed.
3. Then, come errors Connection has been shutdown by transport. Error Code: CONNECTION_IDLE
4. Then, come erors Requesting HTTP version 3.0 with version policy RequestVersionExact while unable to establish HTTP/3 connection. and they are showing till the end.

(also note the numbers that are now significantly bigger than in #55810)

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Last success run was on 7/20
https://dev.azure.com/dnceng/public/_build/results?buildId=1248255&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd&l=700

client_1  | [07/20/2021 14:15:10] Total: 8,238 Runtime: 00:31:00
client_1  | 	 0: GET                       Success: 543	Canceled: 42	Fail: 0
client_1  | 	 1: GET Partial               Success: 539	Canceled: 46	Fail: 0
client_1  | 	 2: GET Headers               Success: 549	Canceled: 37	Fail: 0
client_1  | 	 3: GET Parameters            Success: 532	Canceled: 55	Fail: 0
client_1  | 	 4: GET Aborted               Success: 549	Canceled: 37	Fail: 0
client_1  | 	 5: POST                      Success: 549	Canceled: 37	Fail: 0
client_1  | 	 6: POST Multipart Data       Success: 541	Canceled: 45	Fail: 0
client_1  | 	 7: POST Duplex               Success: 544	Canceled: 42	Fail: 0

(strange that numbers are even lower than reported in #55810)

Regression seen from 7/21
https://dev.azure.com/dnceng/public/_build/results?buildId=1250670&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=1451f5f3-0108-5a08-5b92-e984b2a85bbd
The log is big so it's not showing in UI, you'll need 'View raw log' button

It goes like this:

1. First, there are some errors like Stream aborted by peer (4294967295) (means stream was disposed prematurely) or timeout or some other unexpected cancellation. Last statistic I was able to see before all requests will start to fail

2021-07-21T13:59:56.4253843Z client_1  | [07/21/2021 13:59:56] Total: 511,180 Runtime: 00:21:00
2021-07-21T13:59:56.4254938Z client_1  | 	 0: GET                       Success: 33,852	Canceled: 2,482	Fail: 0
2021-07-21T13:59:56.4255872Z client_1  | 	 1: GET Partial               Success: 33,948	Canceled: 2,386	Fail: 0
2021-07-21T13:59:56.4257107Z client_1  | 	 2: GET Headers               Success: 33,835	Canceled: 2,500	Fail: 0
2021-07-21T13:59:56.4258055Z client_1  | 	 3: GET Parameters            Success: 33,925	Canceled: 2,410	Fail: 1
2021-07-21T13:59:56.4260027Z client_1  | 	 4: GET Aborted               Success: 33,816	Canceled: 2,506	Fail: 14
2021-07-21T13:59:56.4260783Z client_1  | 	 5: POST                      Success: 33,840	Canceled: 2,494	Fail: 2
2021-07-21T13:59:56.4261487Z client_1  | 	 6: POST Multipart Data       Success: 33,839	Canceled: 2,496	Fail: 1
2021-07-21T13:59:56.4262369Z client_1  | 	 7: POST Duplex               Success: 33,914	Canceled: 2,422	Fail: 0
2021-07-21T13:59:56.4263120Z client_1  | 	 8: POST Duplex Slow          Success: 33,880	Canceled: 2,454	Fail: 2
2021-07-21T13:59:56.4263939Z client_1  | 	 9: POST Duplex Dispose       Success: 33,776	Canceled: 2,560	Fail: 0
2021-07-21T13:59:56.4264675Z client_1  | 	10: POST ExpectContinue       Success: 0	Canceled: 2,491	Fail: 0
2021-07-21T13:59:56.4265339Z client_1  | 	11: HEAD                      Success: 33,889	Canceled: 2,443	Fail: 2
2021-07-21T13:59:56.4265994Z client_1  | 	12: PUT                       Success: 33,855	Canceled: 2,477	Fail: 2
2021-07-21T13:59:56.4266650Z client_1  | 	13: PUT Slow                  Success: 33,641	Canceled: 2,691	Fail: 1
2021-07-21T13:59:56.4267299Z client_1  | 	14: GET Slow                  Success: 33,980	Canceled: 2,352	Fail: 1
2021-07-21T13:59:56.4267950Z client_1  | 	    TOTAL                     Success: 473,990	Canceled: 37,164	Fail: 26

2. Then, come errors of aborting because of premature connection close. After that, as mentioned, no new requests would succeed.
3. Then, come errors Connection has been shutdown by transport. Error Code: CONNECTION_IDLE
4. Then, come erors Requesting HTTP version 3.0 with version policy RequestVersionExact while unable to establish HTTP/3 connection. and they are showing till the end.

(also note the numbers that are now bigger than in #55810)

Author:	CarnaViire
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[CarnaViire]

I realized that it's not really a regression, it's more like #55810 got fixed somehow. Before, we had super small numbers of requests and they didn't change since 1st minute of running. Now, we finally have more decent-ish numbers and that surfaced the issues we weren't able to see before.

I wonder if connection getting closed and then not opening again is yet another symptom of #55979

Sadly I'm unable to reproduce the problem (closed connection) locally even though it's the same docker container :( It seems I run out of ports earlier due to #56151, will need to check out after it gets fixed.

[ManickaP]

Triage: no time to investigate yet, Natalia wasn't able to reproduce locally in the same container. If I don't reproduce it either, we might punt it.