Aggregated `queue_messages_published_total` metric violates Prometheus expectations about counters

[michaelklishin]

See #2781 for the background.

Some metrics, e.g. queue_messages_published_total are computed as a basic sum aggregation of samples from a local ETS table, e.g. channel_queue_exchange_metrics. When a channel is closed, its samples are removed from the table,
decreasing the sum. This violates an expectation for counter metrics in Prometheus: they can only increment or stay flat or reset to 0 but not decrease.

The issue is not present when per-object metrics are used: when a channel is closed, all of its metrics go away, which is what the user expects to happen.

This is a side-effect of our quick-and-dirty switch to aggregated metrics. We need to retain this historical total or delegate
to the Prometheus client library which will do most aggregation work and handle resets.

This is node local state, so we can address this even with a significant rework of the Prometheus plugin and still ship it in a 3.8.x release.

Per discussion with @dcorbacho @gerhard @kjnilsson.

[michaelklishin]

References #2512, #2513.

[gerhard]

Will be picking this one up first thing with @mkuratczyk, and working with @dcorbacho on the review so that she can continue focusing on her in-flight.

[dcorbacho]

Luckily we're not tracking totals on the channel but doing increments, see https://github.com/rabbitmq/rabbitmq-server/blob/master/deps/rabbit/src/rabbit_channel.erl#L205

So when queue_exchange_stats are increased we can use that value to increase a new publish counter to track totals per node (or even exchange). Here: https://github.com/rabbitmq/rabbitmq-server/blob/master/deps/rabbit/src/rabbit_channel.erl#L2123

This should be a very simple addition. Tracking this metric per node means that no garbage collection is needed, the new table only needs to be included on the tables to be reset (API command) and ignored everywhere else. The prometheus collector could stop scrapping channel_queue_exchange_metrics when per_object is selected and use the new metric.