Skip to content

Fixes for many of the CI builds. #5496

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
Nov 19, 2020
Merged

Fixes for many of the CI builds. #5496

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
8f02adc
yml log changes
michaelgsharp Nov 10, 2020
145a150
fixing dependency version
michaelgsharp Nov 10, 2020
3dbac07
package restore location
michaelgsharp Nov 11, 2020
d22f89f
package restore location
michaelgsharp Nov 11, 2020
750121b
path testing
michaelgsharp Nov 11, 2020
167f2e7
path testing
michaelgsharp Nov 11, 2020
386884e
path testing
michaelgsharp Nov 11, 2020
ff5bbb2
path testing
michaelgsharp Nov 11, 2020
0e36f7e
path testing
michaelgsharp Nov 11, 2020
6cd138d
path testing
michaelgsharp Nov 11, 2020
811dd12
path testing
michaelgsharp Nov 11, 2020
0a54309
path testing
michaelgsharp Nov 11, 2020
686979e
path testing
michaelgsharp Nov 11, 2020
cc2dd55
path testing
michaelgsharp Nov 11, 2020
0144301
path testing
michaelgsharp Nov 11, 2020
c9a8130
path testing
michaelgsharp Nov 11, 2020
bfdacdc
path testing
michaelgsharp Nov 12, 2020
3d8db4b
path testing
michaelgsharp Nov 12, 2020
40f8ff1
path testing
michaelgsharp Nov 12, 2020
0ad0aea
path testing
michaelgsharp Nov 12, 2020
3ea6f04
path testing
michaelgsharp Nov 12, 2020
072ca3e
path testing
michaelgsharp Nov 12, 2020
16efe30
yml changes
michaelgsharp Nov 16, 2020
d9e1359
changes for x86
michaelgsharp Nov 18, 2020
24fe90a
changes from pr commeonts
michaelgsharp Nov 18, 2020
5d66c07
changes from pr comments
michaelgsharp Nov 18, 2020
aa3a413
chages for x86
michaelgsharp Nov 19, 2020
c960425
Restore to older version of branch
michaelgsharp Nov 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 17 additions & 18 deletions build/ci/job-template.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ jobs:
- ${{ if and( eq(parameters.nightlyBuild, 'true'), eq(parameters.pool.name, 'Hosted Ubuntu 1604')) }}:
- bash: echo "##vso[task.setvariable variable=LD_LIBRARY_PATH]$(nightlyBuildRunPath):$LD_LIBRARY_PATH"
displayName: Set LD_LIBRARY_PATH for Ubuntu and CentOS to locate Native shared library in current running path
- script: ${{ parameters.buildScript }} -configuration $(_configuration) /p:TargetArchitecture=${{ parameters.architecture }}
- script: ${{ parameters.buildScript }} -configuration $(_configuration) /p:TargetArchitecture=${{ parameters.architecture }} /p:TestArchitectures=${{ parameters.architecture }} /p:RestorePackagesPath=$(Build.SourcesDirectory)\packages /p:NUGET_PACKAGES=$(Build.SourcesDirectory)\packages
displayName: Build
- ${{ if eq(parameters.pool.name, 'Hosted macOS') }}:
- task: Bash@3
Expand All @@ -84,14 +84,13 @@ jobs:
script: cd packages;find . -type d -path "*/runtimes/osx-*" -exec rm -rv {} +;find . -type d -path "*/runtimes/win-*" -exec rm -rv {} +;cd ..
displayName: Clean up non-Linux runtime folders of NuGet Packages to save disk space
- ${{ if eq(parameters.buildScript, 'build.cmd') }}:
# TODO: We need to do this. THe packages are restored in a different folder so we need find the correct path.
# - task: PowerShell@2
# inputs:
# targetType: inline
# script: Get-ChildItem -Path '.\packages\*\runtimes\*' -Recurse | Select -ExpandProperty FullName | Where {$_ -notlike '*\win-*'} | sort length -Descending | Remove-Item -Recurse -Confirm:$false -Force
# displayName: Clean up non-Windows runtime folders of NuGet Packages to save disk space
- script: dir /s "bin"
- script: dir /s "$(Build.SourcesDirectory)"
displayName: show bin folder disk usage
- task: PowerShell@2
inputs:
targetType: inline
script: Get-ChildItem -Path '$(Build.SourcesDirectory)\packages\*\runtimes\*' -Recurse | Select -ExpandProperty FullName | Where {$_ -notlike '*\win-*'} | sort length -Descending | Remove-Item -Recurse -Confirm:$false -Force
displayName: Clean up non-Windows runtime folders of NuGet Packages to save disk space
- ${{ if eq(parameters.nightlyBuild, 'true') }}:
- script: $(dotnetPath) restore $(nightlyBuildProjPath)
displayName: Restore nightly build project
Expand All @@ -110,16 +109,16 @@ jobs:
displayName: Clean up useless project
- script: $(dotnetPath) msbuild -restore $(nightlyBuildProjPath) /p:ReferenceTypeForTestFramework="Nuget" -configuration $(_configuration) /p:TargetArchitecture=${{ parameters.architecture }}
displayName: Build Nightly-Build Project with latest package versions
- script: ${{ parameters.buildScript }} -$(_configuration) -runnightlybuildtests
- script: ${{ parameters.buildScript }} /p:Build=false -$(_configuration) /p:TargetArchitecture=${{ parameters.architecture }} /p:TestArchitectures=${{ parameters.architecture }} -runnightlybuildtests /p:RestorePackagesPath=$(Build.SourcesDirectory)\packages /p:NUGET_PACKAGES=$(Build.SourcesDirectory)\packages
displayName: Run Nightly Build Tests
- ${{ if eq(parameters.nightlyBuild, 'false') }}:
- ${{ if eq(parameters.innerLoop, 'false') }}:
- ${{ if and(eq(parameters.runSpecific, 'false'), eq(parameters.useVSTestTask, 'false')) }}:
# TODO: Code coverage needs to be fixed.
- script: ${{ parameters.buildScript }} -configuration $(_configuration) -test -ci #-coverage=${{ parameters.codeCoverage }}
- script: ${{ parameters.buildScript }} /p:Build=false -configuration $(_configuration) /p:TargetArchitecture=${{ parameters.architecture }} /p:TestArchitectures=${{ parameters.architecture }} -test -ci #-coverage=${{ parameters.codeCoverage }}
displayName: Run All Tests.
- ${{ if and(eq(parameters.runSpecific, 'true'), eq(parameters.useVSTestTask, 'false')) }}:
- script: ${{ parameters.buildScript }} -configuration $(_configuration) -ci /p:TestRunnerAdditionalArguments='-trait$(spaceValue)Category=RunSpecificTest' #-coverage=${{ parameters.codeCoverage }}
- script: ${{ parameters.buildScript }} /p:Build=false -configuration $(_configuration) /p:TargetArchitecture=${{ parameters.architecture }} /p:TestArchitectures=${{ parameters.architecture }} -test -ci /p:TestRunnerAdditionalArguments='-trait$(spaceValue)Category=RunSpecificTest' /p:RestorePackagesPath=$(Build.SourcesDirectory)\packages /p:NUGET_PACKAGES=$(Build.SourcesDirectory)\packages #-coverage=${{ parameters.codeCoverage }}
displayName: Run Specific Tests.
- ${{ if and(eq(parameters.buildScript, 'build.cmd'), eq(parameters.useVSTestTask, 'true')) }}:
- task: VSTest@2
Expand All @@ -143,7 +142,7 @@ jobs:
collectDumpOn: onAbortOnly
publishRunAttachments: true
- ${{ if eq(parameters.innerLoop, 'true') }}:
- script: ${{ parameters.buildScript }} -configuration $(_configuration) -test -ci /p:TestRunnerAdditionalArguments='-notrait$(spaceValue)Category=SkipInCI' #-coverage=${{ parameters.codeCoverage }}
- script: ${{ parameters.buildScript }} /p:Build=false -configuration $(_configuration) /p:TargetArchitecture=${{ parameters.architecture }} /p:TestArchitectures=${{ parameters.architecture }} -test -ci /p:TestRunnerAdditionalArguments='-notrait$(spaceValue)Category=SkipInCI' /p:RestorePackagesPath=$(Build.SourcesDirectory)\packages /p:NUGET_PACKAGES=$(Build.SourcesDirectory)\packages #-coverage=${{ parameters.codeCoverage }}
displayName: Run CI Tests.
- script: $(dotnetPath) msbuild -restore build/Codecoverage.proj
displayName: Upload coverage to codecov.io
Expand All @@ -162,16 +161,16 @@ jobs:
displayName: Stage build logs
condition: not(succeeded())
inputs:
sourceFolder: $(Build.SourcesDirectory)/artifacts/log/
contents: '**'
targetFolder: $(Build.ArtifactStagingDirectory)artifacts/log/
sourceFolder: $(Build.SourcesDirectory)
contents: 'artifacts/log/**'
targetFolder: $(Build.ArtifactStagingDirectory)
- task: CopyFiles@2
displayName: Stage test output
condition: not(succeeded())
inputs:
sourceFolder: $(Build.SourcesDirectory)/artifacts/TestResults
sourceFolder: $(Build.SourcesDirectory)
contents: |
**
artifacts/TestResults/**
targetFolder: $(Build.ArtifactStagingDirectory)
- task: CopyFiles@2
displayName: Stage process dump and pdb if any
Expand All @@ -191,5 +190,5 @@ jobs:
artifactName: ${{ parameters.name }} $(_config_short)
artifactType: container
- ${{ if eq(parameters.nightlyBuild, 'false') }}:
- script: ${{ parameters.buildScript }} -pack -ci
- script: ${{ parameters.buildScript }} /p:Build=false -pack -ci /p:RestorePackagesPath=$(Build.SourcesDirectory)\packages /p:NUGET_PACKAGES=$(Build.SourcesDirectory)\packages
Copy link
Contributor

@frank-dong-ms-zz frank-dong-ms-zz Nov 18, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/p:Build=false [](start = 46, length = 14)

so with that command we will skip build? If so maybe we may consider remove -build or even -restore from build script and add -build when we do want to build everything out?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Harish and I were just talking about that. we would then have to remember locally to add those in. I think its easier to have it hardcoded in the CI process, then to have every developer remember to do it.

Do you ahve any suggestions on how to do it so the developer doesn't have to add it? We could create a new build script (maybe ci-build.cmd) that has nothing specified. What would you think of that? @harishsk same question.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, in that case I would prefer keep the build script has nothing specified and add any -option as we need although I don't have strong preference on that.

In reply to: 526429534 [](ancestors = 526429534)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you be ok with checking this in for now so we can unblock people and then change this part as needed when we can discuss it more?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'm ok with that one. Please do update the MicrosoftMLOnnxRuntimePackageVersion in eng/Versions.props as I commented in other comments.

In reply to: 526481968 [](ancestors = 526481968)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems michael has rebased the arcade branch with the master branch directly some hours ago? Since now the 1.5.2 for onnx appears on the git history of arcade branch:
https://github.com/dotnet/machinelearning/commits/arcade

These changes also seem to have caused some merge conflicts on your other PR as well as a much bigger diff than the one found earlier today on your PR:
#5494

I'm unsure if this is intended?

displayName: Build Packages
4 changes: 2 additions & 2 deletions eng/Versions.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<GoogleProtobufPackageVersion>3.10.1</GoogleProtobufPackageVersion>
<LightGBMPackageVersion>2.2.3</LightGBMPackageVersion>
<MicrosoftExtensionsPackageVersion>2.1.0</MicrosoftExtensionsPackageVersion>
<MicrosoftMLOnnxRuntimePackageVersion>1.3.0</MicrosoftMLOnnxRuntimePackageVersion>
<MicrosoftMLOnnxRuntimePackageVersion>1.5.2</MicrosoftMLOnnxRuntimePackageVersion>
<MlNetMklDepsPackageVersion>0.0.0.9</MlNetMklDepsPackageVersion>
<ParquetDotNetPackageVersion>2.1.3</ParquetDotNetPackageVersion>
<SystemDrawingCommonPackageVersion>4.5.0</SystemDrawingCommonPackageVersion>
Expand Down Expand Up @@ -44,7 +44,7 @@
<MicrosoftExtensionsTestPackageVersion>3.0.1</MicrosoftExtensionsTestPackageVersion>
<MicrosoftMLTestDatabasesPackageVersion>0.0.6-test</MicrosoftMLTestDatabasesPackageVersion>
<MicrosoftMLTestModelsPackageVersion>0.0.6-test</MicrosoftMLTestModelsPackageVersion>
<MicrosoftMLTensorFlowTestModelsVersion>0.0.12-test</MicrosoftMLTensorFlowTestModelsVersion>
<MicrosoftMLTensorFlowTestModelsVersion>0.0.13-test</MicrosoftMLTensorFlowTestModelsVersion>
<MicrosoftMLOnnxTestModelsVersion>0.0.6-test</MicrosoftMLOnnxTestModelsVersion>
<SystemDataSqlClientVersion>4.6.1</SystemDataSqlClientVersion>
<XunitCombinatorialVersion>1.2.7</XunitCombinatorialVersion>
Expand Down
5 changes: 3 additions & 2 deletions global.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
{
"tools": {
"dotnet": "5.0.100-rc.1.20452.10",
"dotnet": "3.1.102",
"runtimes": {
"dotnet/x64": ["$(MicrosoftNETCorePlatformsVersion)", "$(MicrosoftNETCore3PlatformsVersion)" ]
"dotnet/x64": ["$(MicrosoftNETCorePlatformsVersion)", "$(MicrosoftNETCore3PlatformsVersion)"],
"dotnet/x86": ["$(MicrosoftNETCorePlatformsVersion)", "$(MicrosoftNETCore3PlatformsVersion)"]
}
},
"msbuild-sdks": {
Expand Down
1 change: 1 addition & 0 deletions src/Native/Native.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@

<Target Name="Pack" />
<Target Name="Restore" />
<Target Name="Test" />


</Project>