examples: Moved CI infos to Makefile.ci

[maribu]

Contribution description

• Decluttered the Makefiles by moving the BOARD_INSUFFICIENT_MEMORY lists from each Makefile to a Makefile.ci
• Optimized the list for use of tools:
  • One entry per line reduces the number of merge conflicts
  • One entry per line allows alphabetical sorting e.g. via Vim's sort command
• Sorted all lists alphabetically

Testing procedure

• Check the diffs
• Let the CI do its thing

Issues/PRs references

#11128, #9965

[maribu]

If this PR gets swiftly merged, I might be able to do the same cleanup for the Makefiles in tests today ;-)

[benpicco]

I like the idea!
But instead of having include Makefile.ci in each Makefile, how about adding -include Makefile.ci to Makefile.include

[miri64]

I'm not sure, what this is trying to accomplish... The clutter is just moved to a different spot. Also: the lists are not just about CI, but especially for BLACKLIST/WHITELIST to warn a user, that the app might not be applicable for the board they are using. So the name might be misplaced.

[miri64]

If this PR gets swiftly merged, I might be able to do the same cleanup for the Makefiles in tests today ;-)

A big build system change like this shouldn't be merged right before feature freeze oO.

[benpicco]

I'm not sure, what this is trying to accomplish... The clutter is just moved to a different spot.

Well the clutter is removed from the main Makefile which is actually read by humans, especially the examples.

With the clutter contained to one file, it can clutter that file up as much as it likes - a 100 line Makefile would be atrocious to read, but when it's contained to one file that only does that, the list can take up as much space as it wants, making merging easier.

[maribu]

A big build system change like this shouldn't be merged right before feature freeze oO.

This PR only moves lists from one file to another and sorts their content. As the order of the entries in the list is of no semantic meaning, this PR doesn't change any semantics. But I wouldn't mind waiting a bit either.

I'm not sure, what this is trying to accomplish...

This is a huge reduction of the pain at least I constantly have with those CI lists. Whenever a PR needs to touch one of the lists, the PR is constantly affected by merge conflicts. By having one line per board, this is significantly reduced.

Also with Makefile.ci consisting of at most three lists with one entry at a line, modifying BOARD_INSUFFICIENT_MEMORY with a script is much easier to do. Didn't you once had to add arduino-leonardo to these lists and experienced the pain of working with theses lists yourself?

[cladmi]

For the definition change at least, all values are still the same here.

I used this to print the variables for all values, then sorted to be able to compare and the sorted values are the same as the reference commit.

for app in $(make info-applications); do make --no-print-directory -C ${app} info-debug-variable-APPLICATION info-debug-variable-BOARD_WHITELIST info-debug-variable-BOARD_BLACKLIST info-debug-variable-BOARD_INSUFFICIENT_MEMORY; done | awk '{ delete a; for (i=1; i<=NF; i++) a[$i]++; n=asorti(a, b); for (i=1; i<=n; i++) printf b[i]" "; print "" }'

https://stackoverflow.com/questions/8802734/sorting-and-removing-duplicate-words-in-a-line/25823667#25823667

BOARD_BLACKLIST and BOARD_WHITELIST are not related to CI.
They are currently compensating for a lack of required features and thus do not belong to a Makefile.ci.

A big build system change like this shouldn't be merged right before feature freeze oO.

This PR only moves lists from one file to another and sorts their content. As the order of the entries in the list is of no semantic meaning, this PR doesn't change any semantics. But I wouldn't mind waiting a bit either.

This is a conceptual change that these definitions must be moved to another entity to define them.

The line based definition and re-ordering is not and could go before.

When doing such a change, it is also good to add some static checks to prevent having the old pattern re-appearing again. Many existing PRs do not follow this scheme and will be merged without being updated.

[miri64]

A big build system change like this shouldn't be merged right before feature freeze oO.

Big in the sense of lines touched ;-). I know that the impact shouldn't be that high.

[miri64]

Also: the lists are not just about CI, but especially for BLACKLIST/WHITELIST to warn a user, that the app might not be applicable for the board they are using. So the name might be misplaced.

What about this point though?

[maribu]

Also: the lists are not just about CI, but especially for BLACKLIST/WHITELIST to warn a user, that the app might not be applicable for the board they are using. So the name might be misplaced.

What about this point though?

How about renaming Makefile.ci to no longer indicate a strong connection to the CI? Something like Makefile.boardlists?

[maribu]

Big in the sense of lines touched ;-)

True. So lets wait with merging then. I could also the time add a second commit to apply the approach to tests in this PR, once the discussion reached an agreement.

[benpicco]

BOARD_BLACKLIST and BOARD_WHITELIST are not related to CI.
They are currently compensating for a lack of required features and thus do not belong to a Makefile.ci.

They could go to Makefile.local then.

[maribu]

I think the name Makefile.local is already used to allow developers to add configurations locally to their system that is not intended to be upstreamed. But I'm not sure if I remember this correctly. I think @cladmi is heavily using the features of locally extending the build system to directly select the correct programmer by the value in the BOARD variable on his testing rig, right?

[smlng]

at least for the BOARDS_WHITELIST and BOARDS_BLACKLIST there is IMHO a better way to improve by using features, see #12423 as an RFC example.

[maribu]

at least for the BOARDS_WHITELIST and BOARDS_BLACKLIST there is IMHO a better way to improve by using features, see #12423 as an RFC example.

I agree that using features is generally a better idea than using the black and the whitelist. Even if some of the features would have to be maintained for every board individually, this would be already hugely less pain - the boards are pretty static while the applications are moving targets. But I honestly don't think that this could be done in a single PR, but rather quite a bunch.

I'd say moving those lists out of the main Makefile and having the lists alphabetically sorted and one board per line is a simple and swift way to significantly lessen the pain those lists cause. And this in no way conflicts with any approaches that reduce the number of cases those lists have to be used. I think it is the opposite: Once the lists are easier to manage with scripts (due to having one board in a row and in "lists-only-files"), it will less effort to clean up those list when features can be used instead.

[smlng]

@maribu I somewhat agree for the BOARDS_INSUFFICIENT_MEMORY list, that moving this into a separate file makes it easier to update the list (also allows to copy this to other apps and tests where the same applies).

For black and white list I would prefer to use the feature mechanism as proposed in #12423.

[maribu]

For black and white list I would prefer to use the feature mechanism as proposed in #12423.

Did you notice that I 100% agree on that? ;-)

I only think that until this goal is reached, having those lists separated out, sorted, and formatted one board a line is beneficial.

[smlng]

Did you notice that I 100% agree on that? ;-)

I did ... my suggestion would be to move in parallel, i.e. moving BOARDS_INSUFFICIENT_MEMORY in a separate Makefile (your PR here) and remove black/whitelists by using features (either extend #12423 or add PRs per feature). Instead of moving the latter too and changing it later again.

[maribu]

I did ... my suggestion would be to move in parallel, i.e. moving BOARDS_INSUFFICIENT_MEMORY in a separate Makefile (your PR here) and remove black/whitelists by using features (either extend #12423 or add PRs per feature). Instead of moving the latter too and changing it later again.

OK. Let's do this. @miri64: This way also the name Makefile.ci becomes reasonable, right?

@kaspar030: #9081 is not concerned about BOARD_INSUFFICIENT_MEMORY, so that way the order in which #9081 and this PR are merged no longer matters, right?

[kaspar030]

The blacklists will become much smaller with #9081, to a point where maintaining them will be a non-issue.

I don't like adding another file, with another include and another place to look for configuration.

I suggest moving the "insufficient memory" list to the bottom (right before the include of Makefile.base), with one line per board, alphabetically sorted. It would reduce the maintanance effort.

BOARD_INSUFFICIENT_MEMORY = \
  foo \
  bar \
  #

[maribu]

Another way of reducing those lists and the maintenance burden could be to list CPU_INSUFFICIENT_MEMORY instead of BOARD_INSUFFICIENT_MEMORY

Nice idea. However, board specific bootloaders are not taking into account. As a result the board with the biggest bootloader will force the decision upon other boards. Also, board specific stuff like number of network interfaces used can increase requirements for some board.

Now having written this, I don't think anymore that this will work :-/

[maribu]

@smlng: Check has been added as requested. It works as expected on my machine, hopefully also on others :-)

[maribu]

Aha: The check also works on travis: https://travis-ci.org/RIOT-OS/RIOT/builds/596637735?utm_source=github_status&utm_medium=notification

I'll rebase to also apply the change for examples/suit_update

[maribu]

I believe the issue Murdock points out is a bug in the CI:

/bin/sh: 1: examples/suit_update: Permission denied

Can someone help? I triggered Murdock now three times. In the first run, every test failed with /bin/sh: 1: ./.murdock: not found. The last two runs showed the message above (1 fail, all other tests pass). I think something about this PR triggers bugs in the CI...

[miri64]

For me, bugging ci@riot-labs.de works most of the time ;-).

[maribu]

May I squash?

[smlng]

tested the sanity check it works for all below examples for now. Other changes look good too, if Murdock is happy, this should be good to go.

Please squash.

[maribu]

@smlng: All green :-)