CI fails on azure 32 bit linux Linux_i386: sudo apt-fast install fails

[timotheecour]

I'm seeing:
nim-lang.Nim (packages Linux_i386) Failing after 58s — packages Linux_i386 failed
in several unrelated PRs (eg #17257 (comment))

eg https://dev.azure.com/nim-lang/255dfe86-e590-40bb-a8a2-3c0295ebdeb1/_apis/build/builds/14075/logs/38

2021-03-10T21:10:24.0269335Z CC: stdlib_assertions.nim
2021-03-10T21:10:24.0288424Z CC: stdlib_dollars.nim
2021-03-10T21:10:24.1410956Z Error: execution of an external compiler program 'gcc -c  -w -fmax-errors=3  -I/home/vsts/work/1/s/lib -I/home/vsts/work/1/s -o /home/vsts/.cache/nim/koch_d/stdlib_assertions.nim.c.o /home/vsts/.cache/nim/koch_d/stdlib_assertions.nim.c' failed with exit code: 1
2021-03-10T21:10:24.1412125Z 
2021-03-10T21:10:24.1412666Z In file included from /usr/include/limits.h:25:0,
2021-03-10T21:10:24.1413573Z                  from /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed/limits.h:168,
2021-03-10T21:10:24.1414497Z                  from /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed/syslimits.h:7,
2021-03-10T21:10:24.1415407Z                  from /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed/limits.h:34,
2021-03-10T21:10:24.1416047Z                  from /home/vsts/work/1/s/lib/nimbase.h:268,
2021-03-10T21:10:24.1416680Z                  from /home/vsts/.cache/nim/koch_d/stdlib_assertions.nim.c:9:
2021-03-10T21:10:24.1417570Z /usr/include/features.h:367:25: fatal error: sys/cdefs.h: No such file or directory
2021-03-10T21:10:24.1418154Z compilation terminated.
2021-03-10T21:10:24.1472789Z ##[error]Bash exited with code '1'.
2021-03-10T21:10:24.1576588Z ##[section]Finishing: Build koch

(refs #3123 (comment))
refs:

• sudo apt-get install g++-multilib https://stackoverflow.com/questions/51765069/fatal-error-sys-cdefs-h-no-such-file-or-directory-while-compiling-openssl-1-1
• libc6-dev-i386 https://itectec.com/ubuntu/ubuntu-fatal-error-sys-cdefs-h-no-such-file-or-directory/

[timotheecour]

https://dev.azure.com/nim-lang/Nim/_build/results?buildId=14079&view=logs&jobId=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&j=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&t=38200048-f2b3-5d1c-03e3-3014d58a3656 shows the 1st place where there's a problem:

instead of:

Starting: Install dependencies (i386 Linux)
==============================================================================
Task         : Bash
Description  : Run a Bash script on macOS, Linux, or Windows
Version      : 3.182.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /home/vsts/work/_temp/372abbb7-b8e2-4be2-a494-3b385cd3c98e.sh
Package: libllvm6.0
Pin: origin "azure.archive.ubuntu.com"
Pin-Priority: 1001
[apt-fast 08:23:46]
[apt-fast 08:23:46]Working... this may take a while.
[apt-fast 08:23:46]

03/10 08:23:50 [NOTICE] Verification finished successfully. file=/var/cache/apt/apt-fast/libgpg-error0_1.21-2ubuntu1_i386.deb

03/10 08:23:50 [NOTICE] Download complete: /var/cache/apt/apt-fast/libgpg-error0_1.21-2ubuntu1_i386.deb

as in https://dev.azure.com/nim-lang/Nim/_build/results?buildId=14063&view=logs&jobId=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&j=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&t=38200048-f2b3-5d1c-03e3-3014d58a3656

we get:

https://dev.azure.com/nim-lang/Nim/_build/results?buildId=14079&view=logs&jobId=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&j=ba7bbfa4-f55c-5c34-6d52-1b6b4edd3f37&t=38200048-f2b3-5d1c-03e3-3014d58a3656

Starting: Install dependencies (i386 Linux)
==============================================================================
Task         : Bash
Description  : Run a Bash script on macOS, Linux, or Windows
Version      : 3.182.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /home/vsts/work/_temp/38167460-f4b0-4c90-96fb-020d4d6a6b8f.sh
Package: libllvm6.0
Pin: origin "azure.archive.ubuntu.com"
Pin-Priority: 1001
[apt-fast 23:14:09]
[apt-fast 23:14:09]Working... this may take a while.
E: Unable to correct problems, you have held broken packages.
[apt-fast 23:14:09]Package manager quit with exit code.
Finishing: Install dependencies (i386 Linux)

note that we should abort right here after E: Unable to correct problems, you have held broken packages. instead of continuing

[alaviss]

note that we should abort right here after E: Unable to correct problems, you have held broken packages. instead of continuing

Hmm, apt-fast should have exited right here...

In bash without set -e, the exit code is based on the last command run, and the last command was apt-fast. So I guess apt-fast didn't error out?

[timotheecour]

In bash without set -e, the exit code is based on the last command run, and the last command was apt-fast. So I guess apt-fast didn't error out?

I think it did but not using set -e is a bug as it keeps going instead of error-ing immediately. Even if the exit code is based on last command, you'd get a false success with:

cmd that fails
cmd that succeeds

as I've observed in #17326, we need to add set -e in the bash section to exit on 1st error; I will do that in upcoming PR to fix this

• I have 1 PR that fixes CI on i386 here: fix #17325 unrbreak CI i386 #17327 by upgrading to ubuntu-18.04' on i386
• I'm curious whether it's possible to keep ubuntu-16.04' on i386, but haven't figured out so far, see refs #17325 #17326

[alaviss]

I have 1 PR that fixes CI on i386 here: #17327 by upgrading to ubuntu-18.04' on i386

I would like to avoid this if possible.

[alaviss]

In the future I'll enginner a docker image as CI for nim, which should avoid all of these stupid issues with apt.

If you want to upgrade the ubuntu version @timotheecour, please upgrade it for all architectures.