Host RPM/DEB in yum/apt repo

[arnetheduck]

In status-im/nimbus-eth2#3034 we create RPM:s/DEB:s as part of the release process - the next step would be for these to be uploaded to a repo that users can add to their package manager and thus have the ability to upgrade using package manager commands.

This would require figuring out how such repos are created and where from they can be served.

[jakubgs]

It appears there already is a script for building DEB and RPM packages using fpm:
https://github.com/status-im/nimbus-eth2/blob/unstable/scripts/make_packages.sh

[jakubgs]

Some good docs on APT repositories:

• https://help.launchpad.net/Packaging/PPA?action=show&redirect=PPA
• https://assafmo.github.io/2019/05/02/ppa-repo-hosted-on-github.html
• https://gist.github.com/kauffmanes/5aa81ee975ea78b2d63761a43be7b63d
• https://earthly.dev/blog/creating-and-hosting-your-own-deb-packages-and-apt-repo/

[jakubgs]

Apparently:

Note: We do not allow uploading pre-built binary packages.

https://help.launchpad.net/Packaging/PPA?action=show&redirect=PPA

[jakubgs]

As far as I can tell PPAs are only for ubuntu, and it would be better to get two birds stoned at the same time by also covering Debian with a normal DEB repository.

[jakubgs]

This page has a good overview of available tools for managing Debian repositories:
https://wiki.debian.org/DebianRepository/Setup

[jakubgs]

Some options I'm considering:

• https://www.aptly.info/
• https://pulpproject.org/
• https://github.com/lkhq/laniakea
• https://github.com/KanoComputing/kano-repository-manager
• https://salsa.debian.org/brlink/reprepro

[jakubgs]

Aptly seems quite robust, and has option to publish your repo to S3, or host it yourself by publishing it locally:

• https://www.aptly.info/doc/feature/s3/
• https://www.aptly.info/doc/feature/filesystem/

And it supports management of signing with GPG: https://www.aptly.info/doc/feature/pgp-providers/

[jakubgs]

This doc shows how easy it is to create a repo: https://www.aptly.info/tutorial/repo/

But my question is how we could tie this to CI, since it seems aptly repo add needs to be run locally.

[jakubgs]

The workflow is pretty comfy:

 > aptly repo create -distribution=all -component=main nimbus-eth2       
Local repo [nimbus-eth2] successfully added.
You can run 'aptly repo add nimbus-eth2 ...' to add packages to repository.

 > aptly repo add nimbus-eth2 ./build/                                   
Loading packages...
[+] nimbus-beacon-node_22.4.0_amd64 added
[+] nimbus-beacon-node_22.5.0_amd64 added
[+] nimbus-beacon-node_22.6.1_amd64 added

 > aptly repo show -with-packages nimbus-eth2
Name: nimbus-eth2
Comment: 
Default Distribution: all
Default Component: main
Number of packages: 3
Packages:
  nimbus-beacon-node_22.6.1_amd64
  nimbus-beacon-node_22.5.0_amd64
  nimbus-beacon-node_22.4.0_amd64

[jakubgs]

The DEBs by default end up in ~/.aptly/pool/:

 > find ~/.aptly/pool/ -type f 
/home/jakubgs/.aptly/pool/64/8b/08597feca8721ee8c44300c2f8bf_nimbus_beacon_node_22.6.1_amd64.deb
/home/jakubgs/.aptly/pool/05/a4/5026b2028551fa838bc2cfc4ae43_nimbus_beacon_node_22.5.0_amd64.deb
/home/jakubgs/.aptly/pool/71/61/ef2a6fd670065bb2227a2c879fb6_nimbus_beacon_node_22.4.0_amd64.deb

While the repo metadata is helf in an LevelDB database in ~/.aptly/db/:

 > ls -l ~/.aptly/db/
total 10
-rw-r--r-- 1 jakubgs jakubgs  306 Jul  1 13:35 000002.ldb
-rw-r--r-- 1 jakubgs jakubgs 2687 Jul  1 13:35 000005.ldb
-rw-r--r-- 1 jakubgs jakubgs    0 Jul  1 13:35 000006.log
-rw-r--r-- 1 jakubgs jakubgs   16 Jul  1 13:35 CURRENT
-rw-r--r-- 1 jakubgs jakubgs    0 Jul  1 13:34 LOCK
-rw-r--r-- 1 jakubgs jakubgs 1775 Jul  1 13:35 LOG
-rw-r--r-- 1 jakubgs jakubgs  428 Jul  1 13:35 MANIFEST-000007

[jakubgs]

The GPG signing setup has two options - internal or gpg - which have the same result, but are handled differently:

internal has better handling for batch operations (passing passphrase using command-line arguments)
internal opens keyring only once, so it asks for passphrase once per aptly run, not every time file is signed
internal only supports “classic” format of keyrings

https://www.aptly.info/doc/feature/pgp-providers/

[jakubgs]

Aptly also has an API which can be used to upload DEB packages and publish them:

• https://www.aptly.info/doc/api/
• https://www.aptly.info/doc/api/files/
• https://www.aptly.info/doc/api/publish/
• https://www.aptly.info/doc/api/packages/

This API is open by default and has no auth options. There is a Python and Ruby library:

• https://github.com/gopythongo/aptly-api-client
• https://github.com/sepulworld/aptly_cli

With a reverse proxy with some basic auth it's probably the best way to publish DEBs via CI.

[jakubgs]

There is an Ansible role, but I'd rather write my own: https://github.com/aioue/ansible-role-aptly

[jakubgs]

One crucial question is how this ties to other teams and products within the org. There are a few options:

• Use one single repo like deb.status.im to publish all possible packages from Nimbus, Waku, Desktop, so on.
  • Confused branding, not great from administration point of view, mixing of server software and user applications.
• Use separate repos for each team, for example: deb.nimbus.team, deb.vac.dev, deb.status.im
  • Clear branding, good from administration point of view, clear separation between different types of software.
• Use separate repos for each product, for example: deb.nimbus.team/nimbus-eth2, deb.nimbus.team/nimbus-fluffy
  • Clear branding, extra administration work required, clear separation but might be messy from user point of view.

Possible stakeholders are @arnetheduck, @oskarth, @iurimatias. Thoughts?