Maintain actual CI to confirm reproducibility of roms across distributions

[tlaurion]

Reproducible builds CI based on Ubuntu docker are not working for a long while now:

@osresearch
https://circleci.com/gh/osresearch/heads

@nemanjan00
https://gitlab.com/nemanjan00/heads-ci/

Does anyone have a working one?

Fedora-29 based one does work, but lacks hooks to build from master commits:
https://gitlab.com/tlaurion/heads

Any help appreciated in fixing any of those problems!

Sent from my Galaxy S3 using FastHub-Libre

[BlackMaria]

my $0.25. circleci probably has some restrictions or limits that we are hitting. I cant verify since I don't have access to the account.

1. Space: it may be related to the amount of disk being used.
2. Threads: tons of threads are spawned when using make -j 4. I would suggest removing the -j 4 and see what happens.
   • IMHO we don't really need it to build multi-thread anyway.

NB I have built master in a loop for a weekend and never hit any deadlocks etc.

[tlaurion]

@osresearch : that would be lovely if fixed.
Meanwhile, my gitlabci ubuntu builder works

[tlaurion]

An temporary fix for this was introduced by @osresearch here, using a specific docker image to have bootsrapping tools encompassed to make builds faster, while not permitting easy audit boostrapping dependencies upgrades that could break reproducibility.

While non-ideal, this works.

[tlaurion]

Good news, builds are thought reproducible again since 6b485ed
Edit: not. As per my own reproducibility tests of March 22 2020, it's still not attained completely (bzImage and hotp-verification still not being reproducible)

Any people experienced enough to propose better CircleCI and GitlabCI configurations to be in tree so that CircleCI and GitlabCI could automatically build for different hosts (docker images) and boards upon passed environment variables? (CircleCI doc)

• docker image (ex: ubuntu:18.04, defaulting to it)
  • depending on host image type (ex: ubuntu), install proper dependencies (apt, dnf, others...)
• board configuration (ex: BOARD=x230, defaulting to it)

The community could then subscribe to Heads project over CircleCI and automatically produce default downloadable board artifacts (x230, qemu) for default host environment (ubuntu), append board ROM sha256sum on osresearch's github release page(if built by osresearch), while other builders CIs could infirm/confirm reproducibility across other docker images for a same commit id+board in some way?

[tlaurion]

@osresearch : CI builds are failing again from CircleCI, since they changed their log retention policy, I guess. If you come up with a trick to only output failing logs from build/logs/* (only logs modified in the last 5 seconds shown on screen?)

[Gusher123]

An temporary fix for this was introduced by @osresearch here, using a specific docker image to have bootsrapping tools encompassed to make builds faster, while not permitting easy audit boostrapping dependencies upgrades that could break reproducibility.

While non-ideal, this works.

@tlaurion @osresearch This temporary fix seems to have broken the CI builds from Gitlab since the januari 15th commit 8e4b109 (musl-cross-make) to Heads.

[tlaurion]

@Gusher123: https://app.circleci.com/jobs/github/osresearch/heads/

[Gusher123]

Thanks. That url did not work for me, but https://app.circleci.com/pipelines/github/osresearch/heads did

[tlaurion]

@osresearch : We still have reproducibility issues on commit ad84c38 on those components:

coreboot image differs:

• 1f8405943fdaf6efea02cc07a7cd2f35020baab67d18712035586f5e50cbdbc3 (CircleCI)
• ef52c4eef87efa866e960c1e8f4531117a2d652a59245efdb8a3369777c26770 (GitlabCI)

Because of:

• bzImage :
  • 795b307945343c5c7eb0b55a9285bb73f96832d295da9be5e11887db804ebe02 (CircleCI)
  • 69961890c7ee425855f8620562b5627584f7b2c5f52132291836445087cc90ef (GitlabCI)
    Rest is reproducible. @osresearch ?

Other non-reproducible component:

• librem-hotp-verification (To be addressed in libremkey-hotp-verification is not reproducible #640 since not part of x230 or other generic, non CONFIG_LIBREMKEY board configs )
  Edit: comment here shows non-reproducibility.

Edit: librem-hotp-verification is reproducible again since 6916069