Skip to content

Commit

Permalink
Merge arcade to master (#5525)
Browse files Browse the repository at this point in the history
* Initial commit for Arcade migration

* Added omitted files

* Changed strong name signing to use the same key for shipping and test assemblies

* arcade linux build (#5423)

* arcade linux build

* put file execution permission change into source control

* The `-test` command for windows. Nuget packages (#5464)

* working on testing

* testing updates

* tests almost working

* build changes

* all tests should be working

* changes from PR comments

* fixes for .net 3.1

* Fixed extension check. Removed <PackageId> where not needed

* Removed pkg folder and updated paths.

* Added test key. (#5475)

* Added test key.

* Update PublicKey.cs

Removed extra newline.

* Update ComponentCatalog.cs

Fixed 3 spaces to 4.

* Windows CI working (#5477)

* ci testing changes

* comments from pr

* Added Linux & Mac changes for Arcade (#5479)

* Initial Windows, Linux, Macos builds test

* Add Linux/MacOS specific CI requirements

* Run Arcade CI tests on MacOS/Linux

* Fix final package building

* Add benchmark download to benchmars .csporj file

* Print detailed status of each unit test

* Install CentOS & Ubuntu build dependencies

* Use container names to differenciate between Ubuntu & CentOS

* Remove sudo usage in CentOS

* Fix Linux build dependencies

* Add -y param to apt install

* Remove installation of Linux dependencies

* Minor additions

* Rename Benchmarks to PerformanceTests for Arcade

* Changes

* Added benchmark doc changes

* Pre-merge changes

* Fixing failing Arcade Windows Builds (#5482)

* Try Windows build single quote fix

* Remove %20

* Added variable space value

* Using variables for spacing

* Added space values as job parameters

* Try conditional variables again

* fix official builds

* Revert "fix official builds"

This reverts commit 7dbbdc7.

* fixing tensorflow rebase issue

* Fixes for many of the CI builds. (#5496)

* yml log changes

* Fix NetFX builds by ensuring assembly version is set correctly and not to Arcade default of 42.42.42.42 (#5503)

* Fixed official builds for Arcade SDK (#5512)

* Added fixes for official builds

* Make .sh files executable

* fix mkl nuget issue

Co-authored-by: Frank Dong <frdong@microsoft.com>

* fix code generator tests failure (#5520)

* Added fixes for official builds

* Make .sh files executable

* fix mkl nuget issue

* fix code generate test fails

* only add necessary dependency

Co-authored-by: Mustafa Bal <5262061+mstfbl@users.noreply.github.com>

* Fixed memory leaks from OnnxTransformer (#5518)

* Fixed memory leak from OnnxTransformer and related x86 build fixes

* Reverting x86 build related fixes to focus only on the memory leaks

* Updated docs

* Reverted OnnxRuntimeOutputCatcher to private class

* Addressed code review comments

* Refactored OnnxTransform back to using MapperBase based on code review comments

* Handle integration tests and nightly build testing (#5509)

* Make -integrationTests work

* Update .yml file

* Added the TargetArchitecture properties

* Try out -integrationTest

* Missed -integrationTest flag

* Renamed FunctionalTestBaseClass to IntegrationTestBaseClass

* Missed rename

* Modified tests to make them more stable

* Fixed leak in object pool (#5521)

Co-authored-by: frank-dong-ms <55860649+frank-dong-ms@users.noreply.github.com>
Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com>
Co-authored-by: Mustafa Bal <5262061+mstfbl@users.noreply.github.com>
Co-authored-by: Frank Dong <frdong@microsoft.com>
Co-authored-by: Michael Sharp <misharp@microsoft.com>
Co-authored-by: Antonio Velázquez <38739674+antoniovs1029@users.noreply.github.com>
  • Loading branch information
7 people committed Dec 3, 2020
1 parent d257b88 commit c2f8ed5
Show file tree
Hide file tree
Showing 321 changed files with 14,109 additions and 1,265 deletions.
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@

# Tool Runtime Dir
/[Tt]ools/
/.dotnet/
/.packages/

# User-specific files
*.suo
Expand Down
6 changes: 3 additions & 3 deletions .vsts-dotnet-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ resources:
containers:
- container: CentosContainer
image: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-8bba86b-20190314145033

- container: UbuntuContainer
image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-mlnet-20200515184230-2c829e8

Expand All @@ -29,7 +29,7 @@ jobs:
_targetFramework: netcoreapp3.1
innerLoop: true
pool:
name: Hosted Ubuntu 1604
name: Hosted Ubuntu 1604

- template: /build/ci/job-template.yml
parameters:
Expand All @@ -38,7 +38,7 @@ jobs:
container: UbuntuContainer
innerLoop: true
pool:
name: Hosted Ubuntu 1604
name: Hosted Ubuntu 1604

- template: /build/ci/job-template.yml
parameters:
Expand Down
142 changes: 46 additions & 96 deletions Directory.Build.props
Original file line number Diff line number Diff line change
@@ -1,117 +1,37 @@
<?xml version="1.0" encoding="utf-8"?>
<Project>
<!-- Directory.Build.props contains the common build settings for all projects in the repo. -->

<Import Project="build/BranchInfo.props" />
<Import Project="build/Dependencies.props" />
<Import Project="Sdk.props" Sdk="Microsoft.DotNet.Arcade.Sdk" />

<Import Project="eng/BranchInfo.props" />
<Import Project="eng/ExternalBenchmarkDataFiles.props" />
<Import Project="eng/TensorflowMetaFiles.props" />

<PropertyGroup>
<CreateSymbolicLinksForPublishFilesIfPossible>true</CreateSymbolicLinksForPublishFilesIfPossible>
</PropertyGroup>

<PropertyGroup>
<Copyright>$(CopyrightNetFoundation)</Copyright>
<TreatWarningsAsErrors>True</TreatWarningsAsErrors>
<DebugType>portable</DebugType>
<DebugSymbols>true</DebugSymbols>
<LangVersion>latest</LangVersion>
</PropertyGroup>

<PropertyGroup>
<Configuration Condition="'$(Configuration)'==''">Debug</Configuration>
<Configurations>Debug;Release;Debug-netcoreapp3_1;Release-netcoreapp3_1;Debug-netfx;Release-netfx</Configurations>
<Platform Condition="'$(Platform)'==''">AnyCPU</Platform>
<TargetArchitecture Condition="'$(TargetArchitecture)' == ''">x64</TargetArchitecture>
<NativeTargetArchitecture Condition="'$(NativeTargetArchitecture)' == ''">$(TargetArchitecture)</NativeTargetArchitecture>
<PlatformConfig>$(Platform).$(Configuration)</PlatformConfig>
<StrongNameKeyId>Open</StrongNameKeyId>
</PropertyGroup>

<PropertyGroup>
<RestoreSources>
https://api.nuget.org/v3/index.json;
https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json;
https://dotnet.myget.org/F/dotnet-core/api/v3/index.json;
https://dotnet.myget.org/F/roslyn-analyzers/api/v3/index.json;
https://pkgs.dev.azure.com/dnceng/public/_packaging/MachineLearning/nuget/v3/index.json;
https://pkgs.dev.azure.com/dnceng/public/_packaging/machinelearning-testdata/nuget/v3/index.json;
</RestoreSources>
</PropertyGroup>

<!-- Common repo directories -->
<PropertyGroup>
<RepoRoot>$(MSBuildThisFileDirectory)</RepoRoot>
<SourceDir>$(RepoRoot)src/</SourceDir>
<PackageAssetsPath>$(ArtifactsDir)pkgassets/</PackageAssetsPath>
<PkgDir>$(RepoRoot)pkg/</PkgDir>

<!-- Output directories -->
<BinDir Condition="'$(BinDir)'==''">$([MSBuild]::NormalizeDirectory('$(RepoRoot)', 'bin'))</BinDir>
<BaseOutputPath Condition="'$(BaseOutputPath)'==''">$(BinDir)</BaseOutputPath>
<ObjDir Condition="'$(ObjDir)'==''">$([MSBuild]::NormalizeDirectory('$(BinDir)', 'obj'))</ObjDir>
<RootIntermediateOutputPath Condition="'$(RootIntermediateOutputPath)'==''">$(ObjDir)</RootIntermediateOutputPath>

<IntermediateOutputRootPath Condition="'$(IntermediateOutputRootPath)' == ''">$(RootIntermediateOutputPath)$(PlatformConfig)\</IntermediateOutputRootPath>
<IntermediateOutputPath Condition="'$(IntermediateOutputPath)' == ''">$(IntermediateOutputRootPath)$(MSBuildProjectName)\</IntermediateOutputPath>
<BaseIntermediateOutputPath Condition="'$(BaseIntermediateOutputPath)' == ''">$(IntermediateOutputPath)</BaseIntermediateOutputPath>

<OutputPath Condition="'$(OutputPath)'==''">$(BaseOutputPath)$(PlatformConfig)\$(MSBuildProjectName)\</OutputPath>

<PackageAssetsPath>$(ObjDir)/packages/</PackageAssetsPath>

<PackageOutputPath Condition="'$(PackageOutputPath)'=='' and '$(NonShippingPackage)' == 'true'">$(BinDir)packages_noship/</PackageOutputPath>
<PackageOutputPath Condition="'$(PackageOutputPath)'==''">$(BinDir)packages/</PackageOutputPath>

<NativeOutputPath>$(BaseOutputPath)$(NativeTargetArchitecture).$(Configuration)\Native\</NativeOutputPath>

<!-- Input Directories -->
<PackagesDir>$(DotNetRestorePackagesPath)</PackagesDir>
<PackagesDir Condition="'$(PackagesDir)'==''">$(RepoRoot)packages/</PackagesDir>
<RestorePackagesPath>$(PackagesDir)</RestorePackagesPath>
<ToolsDir Condition="'$(ToolsDir)'==''">$(RepoRoot)Tools/</ToolsDir>
</PropertyGroup>

<Import Project="$(ToolsDir)BuildVersion.targets"
Condition="Exists('$(ToolsDir)BuildVersion.targets')" />

<!-- Version properties -->
<PropertyGroup>
<VersionPrefix>$(MajorVersion).$(MinorVersion).$(PatchVersion)</VersionPrefix>
<BuildNumberMajor Condition="'$(BuildNumberMajor)' == ''">00001</BuildNumberMajor>
<BuildNumberMinor Condition="'$(BuildNumberMinor)' == ''">0</BuildNumberMinor>
<AssemblyFileVersion Condition="'$(AssemblyFileVersion)'==''">$(MajorVersion).$(MinorVersion).$(BuildNumberMajor).$(BuildNumberMinor)</AssemblyFileVersion>

<StabilizePackageVersion Condition="'$(StabilizePackageVersion)' == ''">false</StabilizePackageVersion>
<IncludeBuildNumberInPackageVersion Condition="'$(IncludeBuildNumberInPackageVersion)' == '' and '$(StabilizePackageVersion)' != 'true'">true</IncludeBuildNumberInPackageVersion>

<VersionSuffix Condition="'$(IncludeBuildNumberInPackageVersion)' == 'true'">$(BuildNumberMajor)-$(BuildNumberMinor)</VersionSuffix>
<NoPackageAnalysis>true</NoPackageAnalysis>
</PropertyGroup>

<!-- SourceLink properties used by dotnet/buildtools - need to be set before importing $(ToolsDir)versioning.props -->
<PropertyGroup>
<UseSourceLink>true</UseSourceLink>
<GitHubRepositoryName>machinelearning</GitHubRepositoryName>
</PropertyGroup>

<!--
Source code control properties used by the .NET Core SDK to inject SCC info into the NuGet package.
In future versions, these will be used for SourceLink and to generate AssemblyInfo.
-->
<PropertyGroup>
<PrivateRepositoryUrl>https://github.com/dotnet/$(GitHubRepositoryName)</PrivateRepositoryUrl>
<PublishRepositoryUrl>true</PublishRepositoryUrl>
<SourceRevisionId>$(LatestCommit)</SourceRevisionId>
</PropertyGroup>

<Import Project="$(ToolsDir)versioning.props"
Condition="Exists('$(ToolsDir)versioning.props') and '$(DisableImportVersioningProps)' != 'true'" />

<!-- Language configuration -->
<PropertyGroup>
<LangVersion Condition="'$(MSBuildProjectExtension)'=='.csproj'">8.0</LangVersion>
<LangVersion Condition="'$(MSBuildProjectExtension)'=='.fsproj'">4.7</LangVersion>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>

<PropertyGroup>
<RunningOnUnix Condition="'$(OS)'!='Windows_NT'">true</RunningOnUnix>
</PropertyGroup>

<!-- Signing properties -->
<PropertyGroup>
<AssemblyOriginatorKeyFile Condition="'$(AssemblyOriginatorKeyFile)' == ''">$(ToolsDir)Open.snk</AssemblyOriginatorKeyFile>
<SignAssembly>true</SignAssembly>
<PublicSign Condition="'$(OS)' != 'Windows_NT'">true</PublicSign>
</PropertyGroup>

<!-- Need to explicitly set these properties for the -netcoreapp3_1 or -netfx configurations becuase they are typically based off 'Debug' or 'Release' configs -->
Expand All @@ -125,4 +45,34 @@
<Optimize>true</Optimize>
</PropertyGroup>

<PropertyGroup>
<!-- Optional: Publish the repository URL in the built .nupkg (in the NuSpec <Repository> element) -->
<PublishRepositoryUrl>true</PublishRepositoryUrl>

<!-- Optional: Embed source files that are not tracked by the source control manager in the PDB -->
<EmbedUntrackedSources>true</EmbedUntrackedSources>

<!-- Optional: Build symbol package (.snupkg) to distribute the PDB containing Source Link -->
<IncludeSymbols>true</IncludeSymbols>
<SymbolPackageFormat>snupkg</SymbolPackageFormat>
</PropertyGroup>

<PropertyGroup Condition="'$(DotNetBuildFromSource)' == 'true'">
<!--
When building using source-build the process is:
- Newtonsoft.Json versions 9.0.1 and 12.0.2 are built by source-build
- Version 12.0.2 is written to Version.props
- Arcade needs to use 9.0.1 so we need to override Version.props value here
-->
<NewtonsoftJsonVersion>9.0.1</NewtonsoftJsonVersion>
</PropertyGroup>

<PropertyGroup Condition="'$(DotNetBuildOffline)' == 'true'">
<!--
Arcade has a special version prop for CodeAnalysis.CSharp in GenFacades
to try to match the version loaded by msbuild. In the offline build, this
is simply the source-built version.
-->
<MsbuildTaskMicrosoftCodeAnalysisCSharpVersion>$(MicrosoftCodeAnalysisCSharpVersion)</MsbuildTaskMicrosoftCodeAnalysisCSharpVersion>
</PropertyGroup>
</Project>
82 changes: 35 additions & 47 deletions Directory.Build.targets
Original file line number Diff line number Diff line change
@@ -1,51 +1,39 @@
<Project InitialTargets="CheckForBuildTools">

<ItemGroup>
<PackageReference Include="Microsoft.VisualStudio.Threading.Analyzers" Version="16.5.132" PrivateAssets="all" />
</ItemGroup>

<Target Name="CheckForBuildTools">
<Error Condition="!Exists('$(ToolsDir)')"
Text="The tools directory [$(ToolsDir)] does not exist. Please run build in the root of the repo to ensure the tools are installed before attempting to build an individual project." />
</Target>

<Target Name="CopyNativeAssemblies"
BeforeTargets="PrepareForRun">
<?xml version="1.0" encoding="utf-8"?>
<Project>
<Import Project="Sdk.targets" Sdk="Microsoft.DotNet.Arcade.Sdk" />

<Target Name="CopyNativeAssembiles" AfterTargets="CopyFilesToOutputDirectory">
<PropertyGroup>
<LibPrefix Condition="'$(OS)' != 'Windows_NT'">lib</LibPrefix>
<LibExtension Condition="'$(OS)' == 'Windows_NT'">.dll</LibExtension>
<LibExtension Condition="'$(OS)' != 'Windows_NT'">.so</LibExtension>
<LibExtension Condition="$([MSBuild]::IsOSPlatform('osx'))">.dylib</LibExtension>
</PropertyGroup>

<ItemGroup>
<NativeAssemblyReference>
<FullAssemblyPath>$(NativeOutputPath)$(LibPrefix)%(NativeAssemblyReference.Identity)$(LibExtension)</FullAssemblyPath>
</NativeAssemblyReference>
</ItemGroup>

<Copy SourceFiles = "@(NativeAssemblyReference->'%(FullAssemblyPath)')"
DestinationFolder="$(OutputPath)"
OverwriteReadOnlyFiles="$(OverwriteReadOnlyFiles)"
Retries="$(CopyRetryCount)"
RetryDelayMilliseconds="$(CopyRetryDelayMilliseconds)"
UseHardlinksIfPossible="$(CreateHardLinksForPublishFilesIfPossible)"
UseSymboliclinksIfPossible="$(CreateSymbolicLinksForPublishFilesIfPossible)">
<Output TaskParameter="DestinationFiles" ItemName="FileWrites"/>
</Copy>

<LibPrefix Condition="'$(OS)' != 'Windows_NT'">lib</LibPrefix>
<LibExtension Condition="'$(OS)' == 'Windows_NT'">.dll</LibExtension>
<LibExtension Condition="'$(OS)' != 'Windows_NT'">.so</LibExtension>
<LibExtension Condition="$([MSBuild]::IsOSPlatform('osx'))">.dylib</LibExtension>

<TargetArchitecture Condition="'$(Platform)' == ''">x64</TargetArchitecture>
<NativeTargetArchitecture Condition="'$(NativeTargetArchitecture)' == ''">$(TargetArchitecture)</NativeTargetArchitecture>
<BinDir Condition="'$(BinDir)'==''">$([MSBuild]::NormalizeDirectory('$(RepoRoot)', 'artifacts', 'bin'))</BinDir>
<NativeOutputPath>$(BinDir)Native\$(NativeTargetArchitecture).$(Configuration)\</NativeOutputPath>

<Platform Condition="'$(Platform)'==''">AnyCPU</Platform>
<PlatformConfig>$(Platform).$(Configuration)</PlatformConfig>
<OutputPath Condition="'$(OutputPath)'==''">$(BinDir)$(MSBuildProjectName)\Debug</OutputPath>
</PropertyGroup>

<ItemGroup>
<NativeAssemblyReference>
<FullAssemblyPath>$(NativeOutputPath)$(LibPrefix)%(NativeAssemblyReference.Identity)$(LibExtension)</FullAssemblyPath>
</NativeAssemblyReference>
</ItemGroup>

<Copy SourceFiles = "@(NativeAssemblyReference->'%(FullAssemblyPath)')"
DestinationFolder="$(OutDir)"
OverwriteReadOnlyFiles="$(OverwriteReadOnlyFiles)"
Retries="$(CopyRetryCount)"
RetryDelayMilliseconds="$(CopyRetryDelayMilliseconds)"
UseHardlinksIfPossible="$(CreateHardLinksForPublishFilesIfPossible)"
UseSymboliclinksIfPossible="$(CreateSymbolicLinksForPublishFilesIfPossible)">
<Output TaskParameter="DestinationFiles" ItemName="FileWrites"/>
</Copy>
</Target>

<Import Project="$(ToolsDir)/versioning.targets" Condition="Exists('$(ToolsDir)/versioning.targets')" />

<!-- Workaround: AssemblyInfo.cs is not embedded -->
<!-- https://github.com/dotnet/sourcelink/issues/572 -->
<PropertyGroup>
<TargetFrameworkMonikerAssemblyAttributesPath>$([System.IO.Path]::Combine('$(IntermediateOutputPath)','$(TargetFrameworkMoniker).AssemblyAttributes$(DefaultLanguageSourceExtension)'))</TargetFrameworkMonikerAssemblyAttributesPath>
</PropertyGroup>
<ItemGroup>
<EmbeddedFiles Include="$(GeneratedAssemblyInfoFile)"/>
</ItemGroup>

</Project>
</Project>
Loading

0 comments on commit c2f8ed5

Please sign in to comment.