Support for buffer_timekeys metric

[stevenjm]

Hi,

I've recently had a buffer_timekeys metric added to fluentd, and I'd like to add support for this to the prometheus plugin.

However, this metric takes the form of a list and I'm not sure how best to represent this as a Prometheus metric. I can see a few options:

1. Add one gauge metric per list item, and delete these every time we refresh the list index. This seems like a hack.
2. Imitate a histogram by calculating a set of quantiles from the list data. In implementation, this would also be a set of gauge metrics, but recalculated and updated periodically.
3. As a special case of the above, only provide the newest and oldest timekeys. This is probably sufficient for most monitoring requirements, but ignores most of the source data.

All of these are straightforward to implement, and I'm willing to raise a PR with an implementation. I'm just interested in first finding out what you think the best approach is.

[kazegusuri]

I'm 100% not sure about the feature. I guess each buffer chunks have created time and the buffer instance have a list of the time.
What use case do you consider to send those information to prometheus?
I think the oldest chunk's created time is only useful. So it's better to have option 3.

3\. As a special case of the above, only provide the newest and oldest timekeys. This is probably sufficient for most monitoring requirements, but ignores most of the source data.

[stevenjm]

The timekey of a chunk is not its creation time, but it represents a time range for the log events it can contain. See https://docs.fluentd.org/configuration/buffer-section#time.

I can see a use case for both the oldest and newest chunks. Monitoring the oldest one gives insight into delays within fluentd itself, while monitoring the newest can give insight into delays upstream of fluentd -- that is, if the newest message fluentd has is 5 minutes old, then whatever is sending logs to fluentd is delayed.

I'm fine with only exposing these two metrics if you think that's best. I'll work on a PR for that if nobody else has any other views.

[stevenjm]

I've raised #97 as a proposed implementation.

[stevenjm]

Implemented by #97.