Support array of strings as a value for tags/labels

[hmdhk]

Description of the issue

Currently tags/labels are stored as key/value pairs in ES, however, we have seen users expecting tags to be stored as an array of strings, this is consistent with ECS-style tags.

I'm proposing to support array of strings for adding tags/labels both on the APM server as well as the agent APIs.

The alternative to this approach is to have both labels for key/values and tags for arrays but since supporting just labels (with support for arrays) is more generic, IMO, we should implement the proposed solution.

An example for JavaScript:

apm.addTags({'teams': ['team1','team2']})

What we are voting on

Is this an acceptable solution for agents/apm server?
cc @elastic/apm-agent-devs , @elastic/apm-server

Vote

Team	Yes	No	Indifferent	N/A	Link to issue
.NET
Go
Java
Node.js
Python
Ruby
RUM
APM Server

[felixbarny]

I don't think that this is compatible with ECS:

{
  "tags": {
    "teams": ["team1","team2"]
  }
}

Not sure about that version:

{
  "labels": {
    "teams": ["team1","team2"]
  }
}

I would argue it's not because it's not a key/value pair. But it's probably debatable if non-scalar values are allowed.

However, this version would be compatible with ECS:

{
  "tags": ["team1","team2"]
}

This version would also solve the specific use case this has been requested for, which is to consistently tag documents (not only in APM) with team names to make access rules based on that.

[hmdhk]

@felixbarny , I think this is the same to what I called "alternative approach" in the issue description. So to be clear the second approach is to support both labels for key/values and tags for string arrays, is that what you meant?

It seems redundant to me to support both, but we definitely can do that as well!

[felixbarny]

@ruflin Do you know about prior art regarding labels with multiple values? Is that something you'd consider legal from an ECS perspective?

[ruflin]

@felixbarny I would consider the following "legal" from an ECS perspective:

{
  "labels": {
    "teams": ["team1","team2"]
  }
}

The reason is that any field in Elasticsearch (Lucene) can be/is an array. Even user.id: [12, 17] would be valid. It does not change the mapping type.

What I would consider "illegal" (😆 ) from an ECS perspective is:

{
  "tags": {
    "teams": ["team1","team2"]
  }
}

This changes the mapping type of ECS.

[simitt]

I agree with @felixbarny and @ruflin that the suggested way of storing tags is not ECS compatible. When the argument is consistency with ECS, I don't see why having both, labels and tags in the Elasticsearch docs should be an issue.
Afaik the problem is that currently the agents offer add_tags (potentially deprecated) for what we treat as labels in the ES documents (from 7.0 on).

[hmdhk]

I'm assuming that we will at some point rename anything with "tags" to "labels" on the agent side, i.e. apm.addTags({'teams': ['team1','team2']}) -> apm.addLabels({'teams': ['team1','team2']}) as discussed here. This would generate the following payload which is valid from ECS perspective:

{
  "labels": {
    "teams": ["team1","team2"]
  }
}

I'm ok with supporting both Tags and Labels as well, however, supporting ECS style tags is a breaking (if we keep the same name e.g. addTags) since the agent API should accept a different data type.

We can take the following path: We add labels to the APM server intake spec with array type support (this is not a breaking change). Agents can use labels in their payloads after that's released, this is a breaking change for the agent since the new agent won't work with older APM servers. APM server would change the tags definition for the intake to array of strings since this is a breaking change for the APM server I assume we can only do this in version 8.0, after this point agents can start accepting/sending ECS style tags to APM server.

[watson]

If I recall correctly, the reason why we renamed tags to labels was that tags in ECS were just an array of values with no keys, whereas our definition of tags was "key: value". For that format, the labels in ECS was a better fit - hence the rename.

If labels support that one label key can have multiple values stored as an array, we could technically add support for it, but I'm not sure what the use-case for our users would be? I find having multiple label values for the same label key confusing 🤔

[simitt]

No suggestion yet, but let's separate terminology and review how agents, server and ES behave at the moment.

Agents and Server currently only process and index key:value pairs where values can be of type string, boolean, integer. Those key:value pairs are currently referred to as tags, in some cases also as labels:

Agents
nodejs:

• apm.setTag, apm.addTags (deprecated)
• apm.setLabel, apm.addLabels

python:

• elasticapm.tag, elasticapm.capture_span(tags={})

ruby:

• ElasticAPM.set_tag

rum:

• apm.addTags

go

• transaction.Context.SetTag

java

• transaction.addTag, span.addTag (deprecated)
• transaction.addLabel, span.addLabel

.Net

• transaction.Tags, span.Tags

APM Server Intake API

• *.context.tags
• metadata.labels

ES

• labels, according to top level labels in ECS

This means that the nodejs and java agent currently offer adding labels while all other agents have an API to add tags. The server only uses the term tags within context, but labels for metadata, describing the same data structure. In ES >=7.0 we store the data as labels, in 6.x as tags.

[watson]

@simitt According to #42, it was my understanding that all agents were supposed to rename tags to labels. This is why we did it in the Node.js agent. The old tags API is just an alias for the new labels API and only kept around for backward compatible reasons (will be removed in the next major)

[eyalkoren]

What @watson said (replace Node.js with Java)

[simitt]

I meant to give a neutral summary of where we are right now, as I found the arguments on what is used where and what would be a breaking change hard to follow. (No need for explanation or defense for already changing to labels or sticking to tags).

[wolframhaussig]

If labels support that one label key can have multiple values stored as an array, we could technically add support for it, but I'm not sure what the use-case for our users would be? I find having multiple label values for the same label key confusing

Maybe this helps:
We have a Java web application which is based on the quartz job engine and those jobs read data from different sources and based on that data generates Jobs to transfer them to different target systems. For performance reasons some usecases do not process single jobs - instead we process batches of jobs. To be able to trace performance problems we are storing the information about the jobs in labels:
The label "job_id" contains a single numeric Job ID for jobs which process one job at a time.
The label "job_list" contains the list of Job IDs for jobs with batch features. As it is currently not possible to store arrays of values we are currently using a String value with the comma separated list of job ids.

Having an array here would be a cleaner solution.

[estolfo]

@webmat We discussed this issue in our agents meeting yesterday. We were wondering if you could comment on the compatibility of the proposed format for labels with ECS?

@simitt Would supporting the format mentioned in this comment entail a lot of work on the server side?