Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experiment with modularizing Travis jobs #2737

Merged
merged 1 commit into from
Feb 27, 2019

Conversation

ceedubs
Copy link
Contributor

@ceedubs ceedubs commented Feb 18, 2019

Resolves #2570

This sets up the Cats Travis build to use Travis's stages feature and
modularizes the Travis jobs. This probably introduces a little bit of
extra overhead since each new job needs to spin up a virtual machine and
pull the cache. However, I think that this extra overhead is offset by a
number of factors (see further below). And in experimenting with builds on
my fork, the build time looks to have gone from ~43 min to ~33 min with this
change. See here for an example.

Benefits

We are now only publishing code coverage reports for the default Scala
version (currently 2.12.x). I don't think that publishing multiple
coverage reports for the same commit even has well-defined results, and
I suspect that we didn't really mean to do that. Since running the tests
with code coverage is probably the most time-consuming thing that the
tests do, this is a significant time saver.

Each individual job has a smaller run duration (usually < 20 min). This
means that if an individual job does fail and we have to restart it, we
are waiting for a smaller amount of time than previously.

In theory at least, there's more ability for parallelization. In testing
on my fork, it looks like Travis tends to run 3 to 5 jobs in parallel.
So since we already had that many jobs before, this may not be much of a
win.

Modularizing the build allowed for a few small installation
optimizations (only installing jekyll for the docs job, only installing
node.js for the JS builds, etc).

Lastly, I'm hoping that this sets the stage to move toward releasing via
sbt-ci-release instead of
requiring a maintainer to awkwardly publish from their personal
computer. This new Travis configuration allows us to run a bunch of
tests/validation in parallel and then if all goes well, move on to a
stage that is responsible for releasing. Since each job has its own time
limit, we can probably avoid previous problems of releasing from Travis
causing builds to take too long.

Implementation Notes

Note that Travis stages have awkward interactions with the build matrix,
so I'm now using YAML anchors (essentially templates) to reduce
duplication rather than the build matrix. This allows for finer-grained
control that I think works out pretty nicely.

Resolves typelevel#2570

This sets up the Cats Travis build to use Travis's stages feature and
modularizes the Travis jobs. This probably introduces a little bit of
extra overhead since each new job needs to spin up a virtual machine and
pull the cache. However, I think that this extra overhead is offset by a
number of factors (see further below).

We are now only publishing code coverage reports for the default Scala
version (currently 2.12.x). I don't think that publishing multiple
coverage reports for the same commit even has well-defined results, and
I suspect that we didn't really mean to do that. Since running the tests
with code coverage is probably the most time-consuming thing that the
tests do, this is a significant time saver.

Each individual job has a smaller run duration (usually < 20 min). This
means that if an individual job does fail and we have to restart it, we
are waiting for a smaller amount of time than previously.

In theory at least, there's more ability for parallelization. In testing
on my fork, it looks like Travis tends to run 3 to 5 jobs in parallel.
So since we already had that many jobs before, this may not be much of a
win.

Modularizing the build allowed for a few small installation
optimizations (only installing jekyll for the docs job, only installing
node.js for the JS builds, etc).

Lastly, I'm hoping that this sets the stage to move toward releasing via
[sbt-ci-release](https://github.com/olafurpg/sbt-ci-release) instead of
requiring a maintainer to awkwardly publish from their personal
computer. This new Travis configuration allows us to run a bunch of
tests/validation in parallel and then if all goes well, move on to a
stage that is responsible for releasing. Since each job has its own time
limit, we can probably avoid previous problems of releasing from Travis
causing builds to take too long.

Note that Travis stages have awkward interactions with the build matrix,
so I'm now using YAML anchors (essentially templates) to reduce
duplication rather than the build matrix. This allows for finer-grained
control that I think works out pretty nicely.
@kailuowang
Copy link
Contributor

This looks really cool! Thanks @ceedubs for taking on this.
I will take a proper look at it next week if no one else beat me to it.

@codecov-io
Copy link

codecov-io commented Feb 18, 2019

Codecov Report

Merging #2737 into master will increase coverage by 0.06%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #2737      +/-   ##
=========================================
+ Coverage   95.14%   95.2%   +0.06%     
=========================================
  Files         365     363       -2     
  Lines        6816    6763      -53     
  Branches      302     291      -11     
=========================================
- Hits         6485    6439      -46     
+ Misses        331     324       -7
Impacted Files Coverage Δ
core/src/main/scala/cats/syntax/monoid.scala 0% <0%> (-100%) ⬇️
testkit/src/main/scala/cats/tests/CatsSuite.scala 33.33% <0%> (-36.67%) ⬇️
core/src/main/scala/cats/syntax/validated.scala 80% <0%> (-20%) ⬇️
core/src/main/scala/cats/arrow/Arrow.scala 85.71% <0%> (-14.29%) ⬇️
core/src/main/scala/cats/Contravariant.scala 87.5% <0%> (-12.5%) ⬇️
core/src/main/scala/cats/syntax/either.scala 86.76% <0%> (-12.5%) ⬇️
kernel/src/main/scala/cats/kernel/Monoid.scala 88.88% <0%> (-11.12%) ⬇️
.../src/main/scala/cats/data/RepresentableStore.scala 83.33% <0%> (-4.17%) ⬇️
testkit/src/main/scala/cats/tests/Helpers.scala 96% <0%> (-4%) ⬇️
...rc/main/scala/cats/kernel/instances/function.scala 96.96% <0%> (-3.04%) ⬇️
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2718d5e...4afe71b. Read the comment docs.

Copy link
Contributor

@kailuowang kailuowang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice. Thanks so much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Get rid of publish-travis.sh
4 participants