Non-detected/delayed sanitycheck failures due to ROM/RAM overflow

[pfalcon]

The issue is demonstrated by #6614 , quoting it:

---

This PR demonstrates "delayed" regression due to growing code size. Some patches grew size of various parts of code. But apparently, we don't run the entire testsuite for each PR, so some cases of samples/tests failing for particular boards due to code/RAM size were "skipped". But later, other PRs which cause samples/tests to build may start to fail, even though these PRs don't contain any changes causing the failure.

The issue can be reproduced on the master branch even without dummy changes like this PR does, using:

scripts/sanitycheck -s tests/bluetooth/shell/test -p arduino_101

[pfalcon]

@nashif : FYI

[galak]

This is not a bug, but a choice about how we run sanity. We are always going to have problems with certain configurations not building because of size and they aren't going to get caught during the development CI. This is why we have nightly builds to show such issues.

[pfalcon]

This is not a bug

Well, we can call it "feature" instead of "bug", the end result is the same - it causes unexpected CI failures for unsuspecting users (for me, "Bluetooth shell" samples "got" broken, while I was working on "socket TLS", which aren't directly related).

This is why we have nightly builds to show such issues.

I keep hearing about it, but never see outcomes of it, hence the questions:

1. How to access results of these nightly builds? (Do we have links to it sprinkled around so people don't miss it?)
2. Did it detect the issue exemplified by [DNM] Delayed sanitycheck test failure due to ROM overflow #6614 (tests/bluetooth/shell breaking)?
3. How did it report this issue (I didn't see anything in zephyr-devel@lists.z.org, and I don't know where else such reports should be sent to get prompt analysis and resolution).

We are always going to have problems with certain configurations not building

Yes, but the situation when CI for all PRs pass, and CI for the entire codebase fails afterwards is ... not ideal, to say the least, and worth attention IMHO. This ticket is opened to report that such cases actually happen, serve as a reference for other users who may be hit by it, and as a starting point for improving the situation. If it's not useful in such roles, it can be closed.

[nashif]

How to access results of these nightly builds? (Do we have links to it sprinkled around so people don't miss it?)

subscribe to the zephyr-builds@ mailing list.

Yes, but the situation when CI for all PRs pass, and CI for the entire codebase fails afterwards is ... not ideal, to say the least, and worth attention IMHO.

Not ideal, true, we build too much stuff and we build even for usecases that are not necessarily supported or practical usecases hence the many failures, but there is no way we can build everything for every PR, thats why we have the daily builds and manage to keep those happy, with a bit of delay though.

[pfalcon]

subscribe to the zephyr-builds@ mailing list.

Thanks @nashif, sorry, forgot to thank you for this hint previously (and I'm subscribed).

Given that there're suggested "workaround", as well as we tighten up ROM/RAM requirements of various tests (I myself submitted a couple of PRs which were merged), let me close this to clean up way for the release bug triaging.