Description
As seen in the latest builds of PRs #4849 and #4846, the CentOS debug and release builds are failing due to out-of-memory issues.
The debug build on Centos_x64_NetCoreApp30 is failing due to insufficient storage during the phase of building packages. The release build on Centos_x64_NetCoreApp30 is also failing due to insufficient storage, but this time this happens when benchmark data is being downloaded.
CentOS debug build output:
2020-02-18T05:02:16.7235083Z Build FAILED.
2020-02-18T05:02:16.7235297Z
2020-02-18T05:02:16.7241364Z /__w/1/s/Tools/dotnetcli/sdk/3.0.100/Sdks/NuGet.Build.Tasks.Pack/build/NuGet.Build.Tasks.Pack.targets(198,5): error : No space left on device [/__w/1/s/pkg/Microsoft.ML.DnnImageFeaturizer.ResNet101/Microsoft.ML.DnnImageFeaturizer.ResNet101.nupkgproj]
2020-02-18T05:02:16.7241930Z /__w/1/s/Tools/dotnetcli/sdk/3.0.100/Sdks/NuGet.Build.Tasks.Pack/build/NuGet.Build.Tasks.Pack.targets(198,5): error : No space left on device [/__w/1/s/pkg/Microsoft.ML.DnnImageFeaturizer.ResNet101/Microsoft.ML.DnnImageFeaturizer.ResNet101.symbols.nupkgproj]
2020-02-18T05:02:16.7242290Z /__w/1/s/Tools/dotnetcli/sdk/3.0.100/Sdks/NuGet.Build.Tasks.Pack/build/NuGet.Build.Tasks.Pack.targets(198,5): error : No space left on device [/__w/1/s/pkg/Microsoft.ML.DnnImageFeaturizer.ResNet50/Microsoft.ML.DnnImageFeaturizer.ResNet50.nupkgproj]
2020-02-18T05:02:16.7242672Z /__w/1/s/Tools/dotnetcli/sdk/3.0.100/Sdks/NuGet.Build.Tasks.Pack/build/NuGet.Build.Tasks.Pack.targets(198,5): error : No space left on device [/__w/1/s/pkg/Microsoft.ML.DnnImageFeaturizer.ResNet50/Microsoft.ML.DnnImageFeaturizer.ResNet50.symbols.nupkgproj]
2020-02-18T05:02:16.7243007Z /__w/1/s/Tools/dotnetcli/sdk/3.0.100/Sdks/NuGet.Build.Tasks.Pack/build/NuGet.Build.Tasks.Pack.targets(198,5): error : No space left on device [/__w/1/s/pkg/Microsoft.ML.Mkl.Redist/Microsoft.ML.Mkl.Redist.nupkgproj]
CentOS release build output:
Downloading from "https://aka.ms/mlnet-resources/benchmarks/digits.csv" to "/__w/1/s/test/data/external/digits.csv" (264,712 bytes).
Downloading from "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KTest240kRows.tsv" to "/__w/1/s/test/data/external/MSLRWeb10KTest240kRows.tsv" (172,985,414 bytes).
Downloading from "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KTrain720kRows.tsv" to "/__w/1/s/test/data/external/MSLRWeb10KTrain720kRows.tsv" (519,712,566 bytes).
Downloading from "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KValidate240kRows.tsv" to "/__w/1/s/test/data/external/MSLRWeb10KValidate240kRows.tsv" (169,246,139 bytes).
/__w/1/s/build.proj(90,5): warning MSB3924: Failed to download file "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KValidate240kRows.tsv". Beginning retry 2 in 5000ms. No space left on device
Downloading from "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KValidate240kRows.tsv" to "/__w/1/s/test/data/external/MSLRWeb10KValidate240kRows.tsv" (169,246,139 bytes).
/__w/1/s/build.proj(90,5): warning MSB3924: Failed to download file "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KValidate240kRows.tsv". Beginning retry 3 in 5000ms. No space left on device
Downloading from "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KValidate240kRows.tsv" to "/__w/1/s/test/data/external/MSLRWeb10KValidate240kRows.tsv" (169,246,139 bytes).
/__w/1/s/build.proj(90,5): warning MSB3924: Failed to download file "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KValidate240kRows.tsv". Beginning retry 4 in 5000ms. No space left on device
Downloading from "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KValidate240kRows.tsv" to "/__w/1/s/test/data/external/MSLRWeb10KValidate240kRows.tsv" (169,246,139 bytes).
/__w/1/s/build.proj(90,5): warning MSB3924: Failed to download file "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KValidate240kRows.tsv". Beginning retry 5 in 5000ms. No space left on device
Downloading from "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KValidate240kRows.tsv" to "/__w/1/s/test/data/external/MSLRWeb10KValidate240kRows.tsv" (169,246,139 bytes).
/__w/1/s/build.proj(90,5): warning MSB3924: Failed to download file "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KValidate240kRows.tsv". Beginning retry 6 in 5000ms. No space left on device
Downloading from "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KValidate240kRows.tsv" to "/__w/1/s/test/data/external/MSLRWeb10KValidate240kRows.tsv" (169,246,139 bytes).
/__w/1/s/build.proj(90,5): error : MSB3923: Failed to download file "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KValidate240kRows.tsv". No space left on deviceBuild FAILED.