Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete build files before packaging and increase build timeout #4377

Merged
merged 3 commits into from
Oct 25, 2019

Conversation

codemzs
Copy link
Member

@codemzs codemzs commented Oct 24, 2019

This reverts commit c922529 and deletes build files before packaging to get rid of out of memory error. It also increases the build timeout to 60 minutes.

@codemzs codemzs requested a review from a team as a code owner October 24, 2019 21:17
@eerhardt
Copy link
Member

Do we know what the issue is here?

@codemzs
Copy link
Member Author

codemzs commented Oct 24, 2019

@eerhardt I see some tests fail consistently on the new machines may be the SKU is different but I think we can revert this change and add a build step before we package that deletes files no longer need to free up disk space this was suggested by @safern

@codecov
Copy link

codecov bot commented Oct 24, 2019

Codecov Report

Merging #4377 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4377      +/-   ##
==========================================
- Coverage   74.66%   74.65%   -0.02%     
==========================================
  Files         883      883              
  Lines      155117   155117              
  Branches    16931    16931              
==========================================
- Hits       115814   115795      -19     
- Misses      34558    34572      +14     
- Partials     4745     4750       +5
Flag Coverage Δ
#Debug 74.65% <ø> (-0.02%) ⬇️
#production 70.21% <ø> (-0.02%) ⬇️
#test 89.56% <ø> (-0.02%) ⬇️
Impacted Files Coverage Δ
test/Microsoft.ML.AutoML.Tests/DatasetUtil.cs 79.06% <0%> (-6.98%) ⬇️
...icrosoft.ML.AutoML/Experiment/SuggestedPipeline.cs 88.17% <0%> (-4.31%) ⬇️
....ML.AutoML/PipelineSuggesters/PipelineSuggester.cs 83.19% <0%> (-4.21%) ⬇️
...rosoft.ML.AutoML/ColumnInference/TextFileSample.cs 59.6% <0%> (-3.32%) ⬇️
src/Microsoft.ML.AutoML/Sweepers/Parameters.cs 83.05% <0%> (-0.85%) ⬇️
...soft.ML.TestFramework/DataPipe/TestDataPipeBase.cs 73.6% <0%> (-0.33%) ⬇️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs 84.7% <0%> (-0.21%) ⬇️
...L.AutoML/TrainerExtensions/TrainerExtensionUtil.cs 86.89% <0%> (+1.74%) ⬆️

@safern
Copy link
Member

safern commented Oct 25, 2019

we package that deletes files no longer need to free up disk space this was suggested by @safern

I just added a commit here d3742e4 to add a step to clean-up before building packages on Windows. Later we can add the step for Unix as the script would be different and didn't want to invest much time on it now.

I did some local investigation. After building, downloading test data and running tests the repo size grows to 16GB, after the clean-up it goes down to 1.5GBs which I believe is enough to build packages without hitting the space issue.

Also I did a test build and it didn't repro anymore, however there where 2 timeouts... I think bumping the timeout to 50mins shouldn't hurt.

@codemzs codemzs changed the title Revert "Move windows build machines to NetCorePublic-Pool. (#4375)" Delete build files before packaging and increase build timeout Oct 25, 2019
@@ -10,7 +10,7 @@ parameters:
jobs:
- job: ${{ parameters.name }}
${{ if eq(parameters.codeCoverage, 'false') }}:
timeoutInMinutes: 40
timeoutInMinutes: 60
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should really be careful about letting our builds take so long. This isn't a huge project, and having an hour CI time doesn't seem great. Maybe someone should investigate what is taking so long in our builds.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eric, I will revert this back after the release. We have a huge backlog of PRs that is threatening the release.


In reply to: 339138790 [](ancestors = 339138790)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed - we need to get unblocked in the short term. I meant to say that someone should investigate it after this gets us unblocked.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect the reason our build times went up is because of newly added TensorFlow tests that each take easily a minute to run. We can try to reduce the dataset size but obviously after this release.

CC: @ashbhandare @harshithapv @bpstark

Copy link
Member

@eerhardt eerhardt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Let's get this in to unblock the CI.

@codemzs codemzs merged commit 7cb9893 into dotnet:master Oct 25, 2019
frank-dong-ms-zz pushed a commit to frank-dong-ms-zz/machinelearning that referenced this pull request Nov 4, 2019
…t#4377)

* Revert "Move windows build machines to NetCorePublic-Pool. (dotnet#4375)"

This reverts commit c922529.

* Add step to cleanup test data before building packages to free up space

* Increase build timeout.
@ghost ghost locked as resolved and limited conversation to collaborators Mar 20, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants