Investigate an http.route tag

[codefromthecrypt]

I'm going to recommend we make a standard tag of http.route, a way to help people make good low-cardinality names or tags from http requests. Main idea is that instead of parsing http.url which can be trouble, allow frameworks to provide http.route which might not always be the same format, but would always be variable and parameter free. It also has a higher likelihood of matching 255 char indexing constraints, which are common on storage backends.

Here are some examples of http route values for the same endpoint:

• vert.x web route "/items/:itemId"
• Play "/items/$id<[0-9]+>"
• Spring WebMVC request mapping "/items/{itemId}"

Until we have something like this, it is almost nonsense to answer some folks who want to push spans into metrics (see #1869). For example, in prometheus, counters etc will reset on each small deviation of tags, a problem far worse than ours wrt cardinality.

Example framework relationships:

• Jersey - response has the template
• ratpack - response has the template
• spring - attribute includes the template
• sparkjava - not yet supported
• play - via tags kamon example

In interop tests, we can't necessarily test for equality of this value as it will be framework specific. However, we can test for cardinality! Ex we make an endpoint with a path variable and execute it against two values of that. Then, we test http.path tag is different while http.template is the same. Ideally the http template is already "washed", but those derived from paths might need normalization. For example, you may need to ensure there's only one leading slash.

Besides a lookup tag, http.template can be used to create a better name for the span. For example, if a template is available, it could be a better choice than the http method. Implementation-wise, this implies a span rename feature, as templates are not always known before-the-fact. That said a primary use case is navigating to other systems like metrics, whether that's directly via the http.template tag or something derived from it.

For example, requests that don't match any template need to be eventually handled. For example, if the http.response_code tag is present and http.template is absent, you might have a heuristic to create a bucketed tag like 'REDIRECT' to share with metrics. Doing so allows bot-like behavior from poisoning metrics. This sort of logic exists in micrometer.

Thanks for input on this, notably @jkschneider and @mikewrighton. CC'ing the "firehose" crowd @bplotnick @kaisen @drolando and @narayaruna, and others interested in collaborating like @ivantopo

[codefromthecrypt]

(PS as a clarification, this is not a propagated tag, rather one sent out-of-band and possibly used as an aggregating dimension for metrics)

[SergeyKanzhelev]

Do you expect span's name be the same as http.template in the most cases? Or they semantically different?

[bplotnick]

I'm not opposed to this, but 1) i'm not sure about the name and 2) i don't know if it's necessary

I'm not sure about the name, because I don't know if a template (assuming we're talking URI templates here) is sufficient. Here's an example:

We have this exact problem with our swagger RPC endpoints. Using the petstore example, GET /pet/{petId} will show up as /pet/1234 in the http.uri. So 1) it is difficult to know that that corresponds to the /pet/{petId} path template, and 2) it is impossible to know that it corresponds to the GET version of that vs the DELETE or POST versions.

We solve the second problem by making the span name equal to the method and the path in pyramid_zipkin. However, this doesn't solve the path parameter problem (the uri template).

The Correct Way to solve this would be to use any identifier guaranteed to uniquely identify the endpoint, which in the swagger case is operationId. I think that this should just be set as the span name as I think @SergeyKanzhelev is alluding to. This is why I'm saying that it is probably not necessary.

[jkschneider]

The Correct Way to solve this would be to use any identifier guaranteed to uniquely identify the endpoint

Effectively I think that is what http.template is in this proposal. Making the parameterized URI the de facto identifier across many frameworks makes your traces/metrics look somewhat uniform across the org (i.e. those services using Swagger and those that are not).

[codefromthecrypt]

The Correct Way to solve this would be to use any identifier guaranteed to uniquely identify the endpoint Effectively I think that is what http.template is in this proposal.

right, probably the fact that a template is present or not could be distracting. For example, "post /foo" as long as it has no variables is also fine (ex not a template). Maybe the name template is not great. The goal indeed is to uniquely identify the endpoint.. there is definitely a secondary goal of it being human readable. For example, a hashed identifier wouldn't be great as some will optionally choose to use this as a span name or a human readable label.

[codefromthecrypt]

Do you expect span's name be the same as http.template in the most cases? Or they semantically different?

What's currently known as http.template is a better span name for most people than something like "get". However, think of the redirect case, it would be "REDIRECT" or something else it would have infinite cardinality. The other issue is that a span can fail before it can know its endpoint. This is behind the heuristic of a tentative name even if you are planning to use template at response time (if you get a response). Reason I hesitated about saying it is semantically the span name is because the name is up to users to decide. This could be a decent choice, but could be added as a tag regardless.

[codefromthecrypt]

I'm not sure about the name, because I don't know if a template (assuming we're talking URI templates <https://tools.ietf.org/html/rfc6570> here) is sufficient. Here's an example: We have this exact problem with our swagger <https://swagger.io/> RPC endpoints. Using the petstore example <http://petstore.swagger.io/>, GET /pet/{petId} will show up as /pet/1234 in the http.uri. So 1) it is difficult to know that that corresponds to the /pet/{petId} path template, and 2) it is impossible to know that it corresponds to the GET version of that vs the DELETE or POST versions.

^^ is a tricky problem, and something to deal with in implementation. In brave, we are thinking about using both the request and response to derive the template, which means you can see what the method is. When you can't derive a template, you don't produce the tag. Ex. if the framework can't tell you, then probably better to not add a tag than add one with infinite cardinality. On URI templates, for better or worse we already know that deviations are used, so probably not best to promise URI template formatting, rather focus on the limited cardinality aspect. Whether or not DELETE or POST is in the template is up to us to decide. Http methods are fixed cardinality, and since a primary consumer of this is metrics, it sort of depends on if the system supports multiple lookup keys or not. One thing that could be easier is this.. It is often easier to strip a method than assume it is always there. We could say that the template or whatever we call it tag does not include the method. In fact this is probably easier for all frameworks I can think of.. Would that help?

[jkschneider]

Just want to make a note that Spring's HandlerMapping.BEST_MATCHING_PATTERN_ATTRIBUTE does not normalize away extraneous slashes.

[ivantopo]

Hello there! Happy to see this moving 😄

Is the scope of this fixed to HTTP-related Spans? I can totally see a similar tag/mechanism working to gather metrics on JDBC call spans and anything else that looks like it (Cassandra driver, etc). This happens to be a pretty common topic when people are starting to instrument their apps.. they usually end up with either too little granularity or a crazy explosion of metrics unless some better naming is manually applied. Anything we can do to provide better default names out of the box is a instant win to all users!

PS: Play Framework does a really nice job at exposing the route templates through tags, that might be worth adding in the description.

[codefromthecrypt]

updated with play info. thx @ivantopo cc @jcchavezs

[codefromthecrypt]

just started on this openzipkin/brave#602