TOOL-10145 linux-pkg rework: main linux-pkg change

[pzakha]

This is the main linux-pkg commit in the linux-pkg rework project (TOOL-9778).

See design document for more information.

I've split this PR into 5 commits to make it easier for reviewing, but those commits are not independent from one another, so they will be pushed together.

Here's a summary of each commit:

1. Each package now publishes artifacts to its own S3 directory. linux-pkg packages that have build dependencies on other linux-pkg packages fetch those dependencies from S3.

2. There are 5 new packages for each linux kernel flavour. The linux kernel packages are different from regular packages because they can be generated from 3 different sources, each way having its own config file. While I've tested that packages can be "built" from all of the 3 sources, for now (on master) we only use the "archive" source. The "archive" source essentially means that we are apt-download'ing debs that are provided by Ubuntu from the ubuntu archive (or rather from our local mirror of the archive: the linux-package-mirror).

3. buildpkg would previously try to do many things at once and the logic was becoming confusing. Previously it would be able to build a package, sync a package with upstream, and generate the initial commit for a new package. With the upcoming change buildpkg will only be responsible of building a package.

4. The logic to sync a package with upstream was also modified. There are now 3 separate commands:

   • checkupdates.sh <package>: Check if a package has updates from upstream.
   • sync-with-upstream.sh <package>: This function has two responsibilities. Let's say for the following example that we are running for the "master" product branch. First it will pull the latest changes from upstream and push them to our repository's "upstreams/master" branch. Second it will attempt to merge the changes from "upstreams/master" into "master", and push the changes to a staging branch for building and testing.
   • push-merge.sh <package>: This last command is intended to be called once building and testing of the package is complete. It will push the staged changes generated by sync-with-upstream to the "master" branch.

5. Remove all the logic that is now obsolete. This includes:

   • the buildlist / updatelist commands, since now each command targets a single package and Jenkins takes care of figuring out which packages must be build/updates via the query-packages.sh command.
   • the build-info package. Given that we do not have a buildlist anymore, the build-info packages are no longer generated.
   • some of the old package lists, which were replaced by new ones.
   • the push-updates.sh command that has been replaced by new logic in commit 4.

Dependencies

See work plan

Testing

See test worksheet

[sdimitro]

A couple miscellaneous questions - overall LGTM, appreciative of the source comments too

[pzakha]

@sdimitro I've posted some fixes and I've also updated the README.