Use Travis CI for automated testing

[per1234]

Purpose:

• A Travis CI build will automatically be triggered every time commits are pushed to MegaCore or a pull request is submitted.
  • Test every board configuration that could affect compilation with every example sketch for every bundled library with every supported version of the Arduino IDE.
  • Test every board configuration that shouldn't affect compilation with the BareMinimum sketch with every supported version of the Arduino IDE.
• Publish a tab separated report of the result of every compilation to a subfolder named after the build number in the MegaCore branch of the CI-reports repository.
  • Due to Travis CI not providing secure environment variables to other parties, reports will not be published to your CI-reports repository when a PR is submitted.
• Send an email when the build fails
• Add a build status image to README.md. Note that this image is configured to show the build status of the avr-100-pin branch of the MCUdude/MegaCore branch. When that branch is merged with the master branch you will need to update the URL.

Files:

• .travis.yml - This file configures the Travis CI build
• travis-ci/arduino-ci-script-arduino-ci-script.sh - this file contains functions called from .travis.yml for common tasks related to Travis CI testing of Arduino projects. It is a subtree of https://github.com/per1234/arduino-ci-script

[MCUdude]

Awesome!

[per1234]

I just spotted a drawback to the subtree. If someone does a manual installation of MegaCore the travis-ci folder will cause the warning message:

Could not find boards.txt in /home/travis/Arduino/hardware/MegaCore/travis-ci. Is it pre-1.5?

This doesn't actually break anything and it won't occur with Boards Manager installations but I've seen this cause concern for users of a different hardware package that generates this warning.

[MCUdude]

You're right. I believe the warning goes away if we move it into the avr folder.

This is how it was solved over at the ATTinyCore repo

Travis is working on the build right now. Will there be a problem if I move the travis-ci folder into into /avr/? Any directories that needs to be changed?

[per1234]

That's correct. Sorry about that. I should have caught it. It only happens when you open and close Boards Manager. I actually just noticed it in the Travis CI log. I'm going to add a check for that warning to my script. I have a function in my script set_board_testing that allows the user to turn on detection of this sort of message during compilation of the sketch or package installation but I currently only have one warning that gets checked for so it's not so helpful yet.

[MCUdude]

No worries, I should have caught it too.

I'm quoting myself:

Travis is working on the build right now. Will there be a problem if I move the travis-ci folder into into /avr/? Any directories that needs to be changed?

[per1234]

It won't affect the current build because that's working with MegaCore at 039be8e. When you move the travis-ci folder you also need to change this line of .travis.yml:
https://github.com/MCUdude/MegaCore/blob/avr-100-pin/.travis.yml#L383
from:

 - source "${TRAVIS_BUILD_DIR}/travis-ci/arduino-ci-script/arduino-ci-script.sh"

to:

 - source "${TRAVIS_BUILD_DIR}/avr/travis-ci/arduino-ci-script/arduino-ci-script.sh"

after you commit that it will start another Travis CI build but the current one will continue. It doesn't like to do a lot of builds at once so it may wait until the current build is finished before starting the next one. You can cancel a build if you don't want to wait.

[MCUdude]

I moved the folder and edited .travis.yml. Just noticed that build 3.1 and 3.2 failed. Any idea what happened here? https://travis-ci.org/MCUdude/MegaCore/jobs/231929339

[per1234]

They were canceled. Do you have the "Auto Cancellation" feature turned on in your Travis CI settings for MegaCore?

[MCUdude]

No, don't think so.. Is there a way to highlight the build errors? Takes some time to scroll through 5000+ lines

[MCUdude]

Wait, I might have canceled them by a mistake when trying to cancel the build job from this PR. seems likely..

[per1234]

Is there a way to highlight the build errors? Takes some time to scroll through 5000+ lines

Yes, it's quite a hassle. That was my motivation for all the work I did on the report system, but they still can't tell you exactly what the error was, only to give some helpful information. I open the raw log and then do a search for "error". I need to do some work to reduce some of the unnecessary verbosity of the logs. Some of the processes are printing a lot more output than is necessary for normal usage. I have a function in the script set_script_verbosity that allows it to be increased for debugging but ordinarily it should show the compilation output and not much more.

Wait, I might have canceled them by a mistake when trying to cancel the build job from this PR. seems likely..

Ok, hopefully that's it. You can always restart a build if you want.

[per1234]

Some information regarding build duration:
The Travis CI documentation doesn't do a good job of explaining how the build process works. There are a variety of "lifecycle" steps you can break a .travis.yml file into:
https://docs.travis-ci.com/user/customizing-the-build/#The-Build-Lifecycle
I had initially thought that some or all of the steps before script happened before the virtual machine snapshot was taken so they would not be repeated during every job. Over time I came to realize this was not the case. Every process is repeated for all 170 jobs of the MegaCore build, including downloading and uncompressing all the versions of the Arduino IDE necessary for that job!

So this explains why, even though I managed to reduce the number of compilations from 23000 to 18000, the build time stayed about the same. This is because I had to add more jobs and every job incurs the overhead of the installation processes. This is why a job that only requires the compilation of 10 small sketches like:
https://travis-ci.org/per1234/MegaCore/jobs/231827314
Still takes 4 minutes. While a job that requires the compilation of 366 sketches like:
https://travis-ci.org/per1234/MegaCore/jobs/231827229
takes 33 minutes.

With the current Travis CI system I don't see any way to solve this issue. However, they announced yesterday that they released a new "Build Stages" feature to beta testing:
https://blog.travis-ci.com/2017-05-11-introducing-build-stages
This should allow the installation to be done only once, which would allow a huge reduction in the MegaCore build time. I believe this feature will also allow me to create unified reports instead of one per job. It's ironic that I spent so much time looking for workarounds for these issues just before they fixed them, even though the initial feature request was 4 years ago. I don't know how long it will take before the feature is moved to production.

[MCUdude]

It seems like there's an issue when building for the ATmega640/1280/2560. Seems to be related to the digitalPinToPCMSK macro. Strange that your travis build didn't caught that.

[per1234]

My build did catch it. I mentioned it a few days ago:
#41 (comment)
It only happens with the 100-pin-avr variant of ATmega640/1280/2560.

[MCUdude]

It's been running all night. Turned out that the PCINT related macros where missing for the ATmega1281/2561. I'm going to push theses soon as the error below is resolved.

I also got this strange error a few times. What's causing this?

https://travis-ci.org/MCUdude/MegaCore/jobs/231972399

[per1234]

When Travis CI calls the publish_report_to_repository function it clones the MegaCore branch of the CI-reports repository, commits the job report, and then pushes it back to the repository. The only thing I can think of is that another job finished at around the same time and pushed its report to the repository after the clone but before the push. I don't see why that would be a problem because the commits are for two separate files. So it's possible Git is being overly cautious and I just need to set the right option. I guess I should do a pull right before the push, even though it was probably less than a second since the clone happened.

What's really strange is that that seems to have hung the job. I would expect that after the push failed it should go on to print the report to the log. If it hung then Travis CI should terminate the job after 10 minutes of no activity. I can't explain why neither of those things happened.

I'll do some research and run some tests to improve the script so this doesn't happen. It's a difficult one to work on because it's very hard to reproduce.

[per1234]

I have updated the script to pull from the remote repository just before pushing. This should solve the issue other than the rare case when the other job happens to push in that very small time frame between the pull and the push going through.

I also added code to pull and then retry the push multiple times if it fails. However, the log you linked above indicates that a failed push will cause the build to immediately fail so that won't do any good yet. I am planning to make some changes to the script to ensure the correct failure behavior of all functions:

• pre-script lifecycle step: if anything goes wrong immediately fail the build
• script lifecycle step: If anything goes wrong complete all steps, then fail the build.
• post-script lifecycle steps: If anything goes wrong complete all tasks, don't fail the build

This is the intended behavior stated in the Travis CI documentation but it seems that it is necessary to configure external scripts correctly to achieve this. I know I have the first two working correctly already.

[MCUdude]

Great! Would be great if you'd add it as a PR

[per1234]

OK, test build in progress:
https://travis-ci.org/per1234/MegaCore/builds/232435609
I will submit the PR as soon as it completes without exposing any new issues with the arduino-ci-script script.

[MCUdude]

Looks very promising! Let's say everything is all right, and I merge the change into the avr-100-pin branch. Is it then possible to merge the avr-100-pin branch into the master, while changing the branch path to master in the script at the same time, without adding the change to the avr-100-pin branch?

[per1234]

Sure is nice to see all those green check marks in a row!

I don't think the branch path is coded into the script anywhere. Where did you see that?

[MCUdude]

I was sure you wrote somewhere that the branch name needed to be changed. Maybe you where referring to the badge in the readme?

[per1234]

Yes, that was the only place that avr-100-pin was hardcoded, and I see you already changed that. So it should automatically handle merging to any branch. Travis CI will build commits for any branch in the repository that has a .travis.yml file in the root. I have been doing all my builds of MegaCore on different branch names so I would notice if I had anything branch specific in the script or .travis.yml.