Networking stress tests moved out of Hosted pool #35011

alnikola · 2020-04-15T16:49:46Z

Fixes #34780

ghost · 2020-04-15T16:49:51Z

Tagging subscribers to this area: @ViktorHofer
Notify danmosemsft if you want to be subscribed.

ghost · 2020-04-15T16:53:00Z

Tagging subscribers to this area: @dotnet/ncl
Notify danmosemsft if you want to be subscribed.

alnikola · 2020-04-15T16:53:22Z

/azp list

azure-pipelines · 2020-04-15T16:53:27Z

CI/CD Pipelines for this repository: coreclr-gc-longrunning coreclr-gc-simulator runtime-coreclr outerloop runtime-coreclr crossgen2 runtime-libraries outerloop runtime-libraries outerloop-windows runtime-libraries outerloop-linux runtime-libraries outerloop-osx runtime runtime-libraries enterprise-linux runtime-libraries stress-http runtime-libraries stress-ssl runtime-live-build runtime-coreclr crossgen2 outerloop coreclr-release-outerloop-nightly sync-runtime-to-mono dotnet-runtime-perf runtime-coreclr crossgen2-composite runtime-jit-experimental

alnikola · 2020-04-15T16:53:50Z

/azp run runtime-libraries stress-http

azure-pipelines · 2020-04-15T16:53:59Z

Azure Pipelines successfully started running 1 pipeline(s).

alnikola · 2020-04-15T16:54:07Z

/azp run runtime-libraries stress-ssl

azure-pipelines · 2020-04-15T16:54:16Z

Azure Pipelines successfully started running 1 pipeline(s).

safern · 2020-04-15T18:37:27Z

So the BYOC unix pools don't have powershell in them. I spoke with @alnikola and the most natural thing is to follow what we do for all our build scripts which is providing a Unix and a Windows build scripts to not require devs to install other dependencies in their machines and to have native support. I helped @alnikola by creating the Unix scripts. PTAL

cc: @eiriktsarpalis

safern · 2020-04-15T18:37:43Z

/azp run runtime-libraries stress-http

azure-pipelines · 2020-04-15T18:37:53Z

Azure Pipelines successfully started running 1 pipeline(s).

safern · 2020-04-15T18:38:00Z

/azp run runtime-libraries stress-ssl

azure-pipelines · 2020-04-15T18:38:10Z

Azure Pipelines successfully started running 1 pipeline(s).

safern · 2020-04-15T20:37:09Z

/azp run runtime-libraries stress-http

azure-pipelines · 2020-04-15T20:37:17Z

Azure Pipelines successfully started running 1 pipeline(s).

safern · 2020-04-15T20:37:18Z

/azp run runtime-libraries stress-ssl

alnikola · 2020-06-17T13:39:28Z

I found the cause of current failures in master and opened the PR #38027
to fix it.

As the matter of fact, that PR can fix only Linux tests. but on NanoServer they fail with actual HttpClient errors detected by stress test application. I see a lot of TaskCanceledExceptions thrown on client container while making calls to server, so it might be an infra issue caused by VM perf degradation. I will check if there are same errors in this PR running on a different queue.

alnikola · 2020-06-17T13:57:13Z

@MattGal I ran commands from http.yml building Linux tests locally on my Windows machine and confirmed that step 6 completes successfully (see below). Since I updated the base SDK image version to .NET 5.0, it now uses Build Engine version 16.7.0 whereas previously it was Build Engine version 16.3, so it could be causing that hang. Let's wait and see if build succeeds on CI (SDK image version update was pushed yesterday)

Step 6/10 : RUN dotnet build -c $CONFIGURATION
 ---> Running in 935dd7d2147a
Microsoft (R) Build Engine version 16.7.0-preview-20310-07+ee1c9fd0c for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  Restored /app/HttpStress.csproj (in 3.24 sec).
  You are using a preview version of .NET. See: https://aka.ms/dotnet-core-preview
  HttpStress -> /app/bin/Release/netcoreapp3.0/HttpStress.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:06.41

alnikola · 2020-06-17T13:59:51Z

/azp run runtime-libraries stress-http

azure-pipelines · 2020-06-17T14:00:00Z

Azure Pipelines successfully started running 1 pipeline(s).

- Windows OS version set to 1809

alnikola · 2020-06-17T15:57:37Z

/azp run runtime-libraries stress-http

azure-pipelines · 2020-06-17T15:57:48Z

Azure Pipelines successfully started running 1 pipeline(s).

alnikola · 2020-06-17T15:57:54Z

/azp run runtime-libraries stress-ssl

azure-pipelines · 2020-06-17T15:58:04Z

Azure Pipelines successfully started running 1 pipeline(s).

alnikola · 2020-06-17T20:08:33Z

@MattGal I fixed container builds on both of Windows and Linux for HttpStress and only Windows build for SslStress. On Linux SslStress container build cannot execute SslStress/run-docker-compose.sh and fails with Permission denied. This command is defined in ssl.yml, so it seems to be happening on the host OS. Could you please check why test runner process doesn't have required permissions?

This is the last issue preventing me from completing this PR. HttpStress tests are currently failing due to actual errors in the scenario and our team will check and prioritize them accordingly, but I have to complete this PR first.

MattGal · 2020-06-17T20:22:33Z

@MattGal I fixed container builds on both of Windows and Linux for HttpStress and only Windows build for SslStress. On Linux SslStress container build cannot execute SslStress/run-docker-compose.sh and fails with Permission denied. This command is defined in ssl.yml, so it seems to be happening on the host OS. Could you please check why test runner process doesn't have required permissions?

This is the last issue preventing me from completing this PR. HttpStress tests are currently failing due to actual errors in the scenario and our team will check and prioritize them accordingly, but I have to complete this PR first.

I'll take a look and see if I have anything useful to note; do note the running user on these build agents has some options to unblock itself (will ping you on Teams about specifics)

safern · 2020-06-17T20:57:01Z

On Linux SslStress container build cannot execute SslStress/run-docker-compose.sh and fails with Permission denied. This command is defined in ssl.yml.

@alnikola we need to fix the permissions of the added script (chmod +x). I will push that now.

I will close the PR and trigger the pipelines manually from a branch to not waste resources (runtime, runtime-live-build, runtime-perf build, etc).

safern · 2020-06-17T21:02:31Z

Test build with chmod +x in the .sh file: https://dev.azure.com/dnceng/public/_build/results?buildId=692516&view=results

safern · 2020-06-17T22:57:05Z

@alnikola since I closed the PR it seems like I can't push to your branch; however it seems like the Linux ssl build is now fixed, just the windows one failing. This is the commit that fixes it so that you can cherry pick it: f3466ad

alnikola · 2020-06-18T10:20:59Z

@safern I repeated chmod x command on my branch and reopened this PR. Additionally, I'd like to clarify that HttpStress Windows build is failing due to some actual issues detected by the stress test. We will investigate it once this PR is merged.

alnikola · 2020-06-18T10:21:35Z

/azp run runtime-libraries stress-http

azure-pipelines · 2020-06-18T10:21:47Z

Azure Pipelines successfully started running 1 pipeline(s).

alnikola · 2020-06-18T10:21:50Z

/azp run runtime-libraries stress-ssl

azure-pipelines · 2020-06-18T10:21:59Z

Azure Pipelines successfully started running 1 pipeline(s).

alnikola · 2020-06-18T14:34:51Z

@MattGal @safern Strangely, but after I merged this into master Windows container builds started failing on docker-compose with a NativeCommandError without reporting any further details on both of runtime-libraries http-stress and runtime-libraries ssl-stress.
Could you please check what is going on? Could it be that docker-compose is not properly configured on some agents?

For the reference, these are pipeline runs I did on this PR right before merging. As you can see, Build HttpStress and Build SslStress completed successfully.

MattGal · 2020-06-18T15:43:05Z

@MattGal @safern Strangely, but after I merged this into master Windows container builds started failing on docker-compose with a NativeCommandError without reporting any further details on both of runtime-libraries http-stress and runtime-libraries ssl-stress.
Could you please check what is going on? Could it be that docker-compose is not properly configured on some agents?

For the reference, these are pipeline runs I did on this PR right before merging. As you can see, Build HttpStress and Build SslStress completed successfully.

Last http-stress run in this PR

Last ssl-stress run in this PR

I'm looking, but maybe we can hold off declaring "some machines are not properly configured" til this has worked a single time in master after your commit :)? All we know is it was tried once and failed.

All this error is telling us is that the thing you invoked wrote to standard error. We get nothing else, and with the way it's being invoked I'm pretty sure that we're not even getting its std err.

Example local ISE docker compose std err:

docker-compose : --build-arg is only supported when services are specified for API version < 1.25. Please use a Compose file version > 2.2 or specify which services to build.
At D:\scratch\compose\somestuff.ps1:10 char:1
+ docker-compose --file "$COMPOSE_FILE" build $BUILD_ARGS.Split()
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (--build-arg is ...vices to build.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

That top line may be missed by console host. Since you log none of the args being passed to the command, so I'd recommend adding logging to what the actual command you're running and all other values we can't see here; a few Write-Outputs can really explain things.

My first guess is that PRs have a different value for BUILD_CONFIGURATION than post-CI

Networking stress tests moved out of Hosted pool

6bded78

alnikola requested review from ViktorHofer, safern and a team April 15, 2020 16:49

Dotnet-GitSync-Bot added the area-Infrastructure label Apr 15, 2020

safern approved these changes Apr 15, 2020

View reviewed changes

davidsh added this to the 5.0 milestone Apr 15, 2020

davidsh approved these changes Apr 15, 2020

View reviewed changes

davidsh added area-System.Net and removed area-Infrastructure labels Apr 15, 2020

davidsh added the test-bug Problem in test source code (most likely) label Apr 15, 2020

ViktorHofer approved these changes Apr 15, 2020

View reviewed changes

Provide Unix scripts for stress builds

3659662

Fix build

1f87033

- TFM set to .net5.0

a130d74

- Windows OS version set to 1809

safern mentioned this pull request Jun 17, 2020

Stop ignoring **/_.pdb files in Docker containers #38027

Closed

safern closed this Jun 17, 2020

Chmod +x run-docker-compose.sh

19a9429

alnikola reopened this Jun 18, 2020

alnikola merged commit 3ace198 into dotnet:master Jun 18, 2020

alnikola mentioned this pull request Jun 18, 2020

Fix HTTP/2 race condition with concurrent window update #38065

Merged

BruceForstall mentioned this pull request Jul 8, 2020

AzDO "runtime-libraries stress-http" pipeline failures #38944

Closed

ghost locked as resolved and limited conversation to collaborators Dec 9, 2020

Networking stress tests moved out of Hosted pool #35011

Networking stress tests moved out of Hosted pool #35011

Uh oh!

Conversation

alnikola commented Apr 15, 2020

Uh oh!

ghost commented Apr 15, 2020

Uh oh!

ghost commented Apr 15, 2020

Uh oh!

alnikola commented Apr 15, 2020

Uh oh!

azure-pipelines bot commented Apr 15, 2020

Uh oh!

alnikola commented Apr 15, 2020

Uh oh!

azure-pipelines bot commented Apr 15, 2020

Uh oh!

alnikola commented Apr 15, 2020

Uh oh!

azure-pipelines bot commented Apr 15, 2020

Uh oh!

safern commented Apr 15, 2020

Uh oh!

safern commented Apr 15, 2020

Uh oh!

azure-pipelines bot commented Apr 15, 2020

Uh oh!

safern commented Apr 15, 2020

Uh oh!

azure-pipelines bot commented Apr 15, 2020

Uh oh!

safern commented Apr 15, 2020

Uh oh!

azure-pipelines bot commented Apr 15, 2020

Uh oh!

safern commented Apr 15, 2020

Uh oh!

alnikola commented Jun 17, 2020

Uh oh!

alnikola commented Jun 17, 2020

Uh oh!

alnikola commented Jun 17, 2020

Uh oh!

azure-pipelines bot commented Jun 17, 2020

Uh oh!

alnikola commented Jun 17, 2020

Uh oh!

azure-pipelines bot commented Jun 17, 2020

Uh oh!

alnikola commented Jun 17, 2020

Uh oh!

azure-pipelines bot commented Jun 17, 2020

Uh oh!

alnikola commented Jun 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MattGal commented Jun 17, 2020

Uh oh!

safern commented Jun 17, 2020

Uh oh!

safern commented Jun 17, 2020

Uh oh!

safern commented Jun 17, 2020

Uh oh!

alnikola commented Jun 18, 2020

Uh oh!

alnikola commented Jun 18, 2020

Uh oh!

azure-pipelines bot commented Jun 18, 2020

Uh oh!

alnikola commented Jun 18, 2020

Uh oh!

azure-pipelines bot commented Jun 18, 2020

Uh oh!

alnikola commented Jun 18, 2020

Uh oh!

MattGal commented Jun 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alnikola commented Jun 17, 2020 •

edited

Loading

MattGal commented Jun 18, 2020 •

edited

Loading