Rework Agent containers build

[webbnh]

The current build procedure for the Agent containers no longer works. The first problem is that it insists upon cross-checking all of its default build targets, including Fedora 32 (even if the requested build doesn't include F32...), which doesn't work now that we've removed our support for Fedora 32 from COPR and COPR itself has deprecated Fedora 33 support. The second problem is that its build targets are specific to the respective target platforms, and so adding a new platform requires cookie-cutter copying the existing targets to create targets for the new platform.

This PR makes the following changes:

• Replace the old cookie-cutter approach with generic templates and pattern rules. With this change, the Makefile no longer refers to specific platforms or targets except in the default list of targets to build.
• Switch from docker.io to quay.io as the source for base images. Since the repositories have different names, and since the tags have different formats, this required some changes to the plumbing.
• Update the default list of targets to Fedora 34, Fedora 35, and CentOS 8 -- dropping Fedora 32, Fedora 33, and CentOS 7.

Note that there is nothing intrinsic to these changes which would prevent Fedora 32 or Fedora 36 from working, but we don't currently provide the required stuff on COPR for these targets. Likewise, there is nothing in these changes which prevents CentOS 7 or CentOS 9 from working -- the CentOS 7 build had stopped working prior to these changes (and its build now fails in exactly the same way with these changes), and packagecloud.io, from which we source the Prometheus RPMs, does not yet support CentOS 9.

[I've tagged @portante as a reviewer for, oh, so many reasons; @ndokos, I would appreciate it if you could be my second reviewer, and anyone else is welcome to step up.]

[dbutenhof]

I'm impressed. It's still "Draft" so I won't bother to approve, but it seems to be looking rather comprehensive assuming testing proves it all out in practice.

[dbutenhof]

The make magic is kinda blinding, but it looks good.

[ndokos]

A tour-de-force of makefileology! Opaque but impressive.

A couple of questions:

• Is it possible to produce the Makefile after all the automatic expansions have happened? That might be a good thing to have if one needs to debug it.
• I wonder if some of the opaqueness could be alleviated by using a more expressive language, e.g. by having a program that would write a makefile for some distro (e.g centos-8) that the main Makefile could then run.

In general, I like programs that write programs, but I'm not sure how I feel about Makefiles that create Makefiles (particularly if they don't write them out) :-)

Nevertheless, once the CentOS problems are resolved, I'm ready to approve.

[ndokos]

LGTM

[webbnh]

Is it possible to produce the Makefile after all the automatic expansions have happened? That might be a good thing to have if one needs to debug it.

You can have Make dump its database (which is pretty voluminous) by using the -p flag, which yields all the rules, dependencies, environment variable values, and the normal variable assignments after building whatever target you specify or default to.

Also, using make --trace is nice, as it tells you why Make is performing each step (i.e., it lists the targets as it's building them; --no-silent also prints the recipe commands as they are executed, but it omits the target.

If you're really interested, you can use make -d (or make --debug=<flag-values>) to have Make share its thought processes as it proceeds, but the output is quite voluminous (even compared to the -p output), and usually --trace will suffice.

I wonder if some of the opaqueness could be alleviated by using a more expressive language, e.g. by having a program that would write a makefile for some distro (e.g centos-8) that the main Makefile could then run.

This is certainly a valid approach, but it's not one that I'm partial to. It might be easier to debug, and it might be easier for someone with limited facility with Make to deal with, but, when things are working, it's a more cumbersome solution which often requires more dependencies and ends up being more fragile. Make is intended to be able to handle things like this without resorting to external super- or sub-structures (although, Make is quite capable of make'ing Makefiles (when they are missing or out of date) and then invoking them all from a single make command line...), so, if we can stomach it, I would prefer to go with a single tool with a single input (ignoring all the helper scripts that we've chosen to employ...).