Add option to fetch timestamp from JSON

[fatpat]

This fixes #80

[roidelapluie]

This seems a good idea, however I would prefer to have a fixed format rather than guessing it from a third party library.

[fatpat]

This seems a good idea, however I would prefer to have a fixed format rather than guessing it from a third party library.

OK I'll check

[roidelapluie]

The idea would be to use func Parse(layout, value string) (Time, error) where layout would be a config option.

[rustycl0ck]

Also update the Readme and the example configs for reference.

And link to this section explaining the behavior that when time series are missed for more than 5 minutes, they just "disappear" from graphs, (so that users understand the expected behavior when the scrapes are successful, but metrics are still missing from prometheus):

If no sample is found (by default) 5 minutes before a sampling timestamp, no value is returned for that time series at this point in time. This effectively means that time series "disappear" from graphs at times where their latest collected sample is older than 5 minutes or after they are marked stale.

Staleness will not be marked for time series that have timestamps included in their scrapes. Only the 5 minute threshold will be applied in that case.

[rustycl0ck]

I don't really know how prometheus handles time series with timestamps from exporters. But one of the concerns I initially had when I saw this PR the first time, was that just by looking at Prometheus is it possible for a user know if the target is up or not?

Does prometheus actually somehow show that the last scrape was at time 10 seconds ago, but the actual data collected was with a timestamp of 4 minutes ago. Because if that is not possible, then prometehus can never reliably tell me about the up status of the target.

Would be great if @roidelapluie you an explanation or a link to some blog/doc explaining this. I found this but this is not targeted at timeseries with timestamps, just for missed scrapes which then get marked as stale.

I had put off reviewing this PR for a day when I would actually be able to test this behavior and understand the consequences, but I think that might still take a few weeks before I can afford to do that.

[roidelapluie]

I expect the timestamp to be exported as a metric, not to be set on metrics. I did not review the PR yet, just scanned it very quickly. We should not set timestamps on metrics

[rustycl0ck]

I expect the timestamp to be exported as a metric, not to be set on metrics.

Ah yes, in that case all would be fine. Exporting the timestamp as a metric (probably with unix epoch time as the value) would be fine. Although on initial thought I can't think how such a metric might actually be useful. But I'll let @fatpat decide if this is useful to them or not.

[roidelapluie]

Oh, indeed, the current implentation exposes a metric with a timestamp, which is something we should not do I think.

cc @SuperQ do you agree?

[SuperQ]

@roidelapluie Agreed, it should not expose with a timestamp.