meta-tags (previously known as extrinsic tags)

[shalstea]

Metrics 2.0 supports adding meta data to metrics, however this is at the cost of network bandwidth. A lot of meta data could be very static (e.g. the data-center a machine is in). It would be very nice to have a means for bulk-loading / updating static meta-data and having it merge in with tags.

For example every metric might have a tag host. Associated with the host is a collection of static data, cluster, data-center, os, os-version, etc. We would like to feed this in. From grafana this would appear as tags to the metric.

[Dieterbe]

these are all known issues.

1. all metadata currently transmitted (especially mdm format) is very redundant and too resource intensive (in network bandwith but also (de)serializing overhead in form of cpu time and memory allocations)
2. any metadata should thus be able to be sent/maintained asynchronously from the data stream (see metricdefinition refactor #199 for some concrete ideas on how to address)
3. this applies for both intrinsic and extrensic properties (tags that affect the metric Id and tags that don't)
4. as for exposing to grafana, I see 2 main ways : A) extending the query language to support tag based searching/filtering etc. B) a custom datasource for metrictank that is a superset of the graphite datasource, but with the extensions for displaying tags in the editor etc.

related : #352

[shalstea]

We would love to be able to add detailed descriptions of metrics, with some way to access this if clicking on some visual indicator on a panel that uses the metric. Users would then be able to really understand what the metric means.

Other static data would include things like:
units, graph as rate, etc.

[Dieterbe]

you're describing a feature request for grafana. maybe @daniellee or @torkelo can advise where to direct that topic.
I believe this at least partially overlaps with grafana/grafana#1153

[TheStigB]

Some additional details about the additional meta data / extrinsic tags.

The goal is to provide better filtering and group by capabilities in Grafana / MetricTank, by being able to augment the core tags with additional tags / meta data that should work like any first class tags in Grafana.

So we would like to be able to upload a set of tags that refer to a core tag on a daily or weekly basis.
Example:
Primary Tag, Additional Tags
HostId, Rollout Stage, OS, Location, .......
100, S1, RHEL7.1, DataCenter1

Initially it's ok if we only have one version of the extrinsic tags, having historical versions and handle changes in them overtime is a nice to have. But not a requirement at this point.

[Dieterbe]

The big question here I think is how do we want to do the associations?
Am I reading this right that you'd like to assign the additional tags by hooking them to pre-existing tags?
for example assigning OS and location tags to metrics, by specifying a hostId tag, and then all metrics with that hostId should get the tag?
implementation wise it could take the form as a set of dynamic high-level rules that we take into account when we query the index, or we could actually go and apply all these tags to every single metric.

looping in @DanCech we've previously discussed this. but i don't remember the outcome.

[TheStigB]

You got it. Basically a "table join" on a given tag the provided (per metric point) tags and the additional tags.

As a future enhancement it might be interesting to have two primary tags, so we can limit additional tags to a given namespace. (No need to do this for MVP)

As for "the upload" probably the easiest to via special message / topic over kafka. I don't think one upload needs to be atomic, but individual "key tag vale" and "additional tags" can be handled independently from the other tag values.

[DanCech]

After some lengthy discussions, we came up with a concept that may work for this functionality.

We would start by adding a separate index to hold meta-records that map tag queries (used to identify target series) with a set of "extrinsic" tag/value pairs to be added to those series.

We would then add a second reverse index to allow looking those records up by tag & value, in the same way as the existing reverse index is used to look up series.

The existing index would be augmented by adding a list of the meta-records associated with each series.

When altering the meta-records the system would look up the associated series (by executing the tag queries against the primary reverse index) and update their lists.

When adding a series, it would be compared against each entry in the meta-index to build the list.

We may also want to maintain in the meta-index a list of all series that are associated with each record, since we already have to do the work to produce the lists of meta-records associated with each series. This would be a cost in terms of index size, but would potentially be a big performance boost at query time (see below).

When executing a query, there are quite a few complex scenarios that would need to be dealt with, mostly around how to deal with query conditions. Basically we would need to pick a query condition that requires a non-empty value (as we do already), then do a lookup for that condition in both reverse indexes. Series matched from the primary index would be added to the prospective result set as normal, while any results from the lookup in the second reverse index would be used to look up series associated with the meta-records (either by executing the tag queries against the main reverse index or by getting a list of matching series directly if they were stored in the meta-record), and those would also be added to the prospective result set.

At this point each entry in the result set would need to be "enriched" with the tags from the associated meta-records by walking the list of meta-records in each series (we need to determine how conflicts would be handled when a meta-record contains extrinsic tag values that conflict with other meta-records associated with the series or with intrinsic tags), then we would be able to apply the rest of the query conditions to filter the enriched result down to the final set of series.

[shanson7]

For a concrete use case, we will have tags like datacenter that will match millions of series and only have a couple values. In this case, we would need to have a list of thousands of host values that map to a particular dc.

As an alternative implementation, it could be possible to send the extrinsic tags along with the normal tags, but not use them to calculate the series id. In that case, metrictank would just need to update the index when a full MetricData message is received with different extrinsic tags. If they don't change then the MetricPoint optimization can still be used. This would mean that the application of extrinsic tags would be handled externally to MT.

[shanson7]

Thinking about this a little this morning, I've got the following notes:

1. Memory usage - The memory idx is currently about 60% of our heap usage. In our use case, every extrinsic tag we add will match almost every series. That means giant sets of metric ids.
2. Cost of Update - We will likely do updates every few hours. Having to run every series through an expression match could be quite costly in this scenario, especially seeing as most of them are unlikely to change.

In our particular use case, what we actually need is a mapping from one tag key/value to another. As alluded to in the primary post, most of ours will be keyed off of the host tag. Almost every series has a host tag, and there are many orders of magnitudes more series than there are hosts. With this in mind I propose that we allow a simple mapping of intrinsic tag to extrinsic tags.

That way we could upload something like dc=dc1 maps to host=[host1,host3,host6,...].

At query time, we determine if the expressions references an extrinsic tag and do an efficient lookup (likely map) to determine if the current series mapped key matches the requested. e.g. if someone asks for something like name=abc AND dc=dc1 if we find abc;host=host1 and abc;host=host2 (using the name=abc filter) we can quickly lookup extrinsic_tags["dc"]["dc1"]["host1"] and extrinsic_tags["dc"]["dc1"]["host2"] to find that only host matches.

1. Memory Usage - With this approach, each extrinsic tag just needs the set of mapped intrinsic tags that match. This is a much smaller set in our case (about 20k vs 400M).
2. Cost of Update - If we require that the entire set be updated at once (i.e. all mappings defined for dc=dc1 must be supplied for each update, then it's pretty efficient as a map insert/update.

[Dieterbe]

from here on, this ticket is about the meta tags. the other tangential ideas (such as uploading generic info for display only but not for searching/filtering) can be done in a new ticket.

we're in the design phase for this feature. @replay can you share your work-in-progress design doc.

[replay]

This is a status update where I describe where we are at:
Last week I started working on the implementation as we've planned in the design doc.
So far I wrote the API calls to add and modify the rules which define the associations between meta-tags (extrinsic) and metric-tags (intrinsic), that's pushed in this branch: #960

Over the course of this and next week I'll implement the procedures to update the new index data structures when meta-records get added/modified/deleted and when metrics get added/deleted.
Querying the new index will be the last part, I suspect it will also be the most complicated part.

@shanson7 I would like to come up with an estimate how much additional memory those new data structures are going to consume, I will already take the current plans to optimize memory efficiency into account (@robert-milan is working on that). Could you maybe provide some numbers of your typical planned use cases of the meta tag index? Such as:

• Number of series per MT
• Average number of metric/intrinsic tags per series
• Total number of meta tag rules defining associations between meta tags and metric/intrinsic tags
• Average number of meta tags per series