Badger storage plugin: query service to support spanKind when retrieve operations for a given service.

[guo0693]

Requirement - what kind of business use case are you trying to solve?

Related to #1920

Update badger reader and writer such that we can retrieve operations by service name and span kind

Problem - what in Jaeger blocks you from solving the requirement?

Proposal - what do you suggest to solve the problem or improve the existing situation?

Any open questions to address

[burmanm]

I'll add notes here so that I don't forget (or if someone else is implementing):

• A single index key is enough for serviceName, operationName + span.Kind indexing

Thus, replace the following code from reader:

		if query.OperationName != "" {
			indexSearchKey = append(indexSearchKey, operationNameIndexKey)
			indexSearchKey = append(indexSearchKey, []byte(query.ServiceName+query.OperationName)...)
		} else {
			indexSearchKey = append(indexSearchKey, serviceNameIndexKey)
			indexSearchKey = append(indexSearchKey, []byte(query.ServiceName)...)
		}

Instead, write a single key with operationIndexKey and in the form of query.ServiceName+query.OperationName+tags.span.kind. Then, in the reader scanning, take into account the length of the searched index key such as query.ServiceName has total length of serviceName+overhead+timestamp+sizeOfTraceID and not more. This can be added to the scanFunction to skip keys which do not match here (or stop searching if we bypassed the match).

In the cache.go, remove two full-table-scans and replace with one which would warm the cache for both services, operations as well any theoretical operations+kind query.

Bonus is one less key written to the db with a slight increase in complexity for reading.

[guo0693]

Not familiar with badger, but should the key be query.ServiceName+tags.span.kind+query.OperationName because we want to get a list of operation names given the service name & span.kind?

[burmanm]

It slightly depends. There's two different use-cases for that information, fetching the list of operations vs fetching the list of traces. For traces, is the span.kind used or not? You should optimize for that use-case and then the "list of operations" for UI purposes is done with the cache.go and a table scan of possible pairs of span.kind + operations meaning it doesn't really matter in which order they are.

To clarify, the list of available operations is done once in the startup and never again before a restart. So don't optimize for that, but optimize for the use-case of traces-fetching where the order for smaller TraceID scanning is more important.

[guo0693]

Make sense. I didn't know it's also used for fetching traces.

[Manik2708]

As far I could understand, operations are fetched only from cache not from the store! So do we need to change the cache only? @yurishkuro

[yurishkuro]

Cache loads from the store, to the store needs to support this

[Manik2708]

Cache loads from the store, to the store needs to support this

Got it! Thinking of changing the service key to serviceName+span.kind and operations to serviceName+span.kind+operationName. Will it be viable?

[yurishkuro]

Probably. As long as the list can be scanned.

[Manik2708]

Cache loads from the store, to the store needs to support this

Got it! Thinking of changing the service key to serviceName+span.kind and operations to serviceName+span.kind+operationName. Will it be viable?

There is a problem in this approach! For getting traces, span.kind is always to be passed in the Query as a tag and fetching the spans of all kind will be impossible without multiple iterations and scans. The approach given in this: #1922 (comment) is also leading to this problem. I could think of this solution:
Creating an additional operationNameWithKindIndexKey with span.kind instead of embedding them in existing keys. When in tags "span.kind" would be present we can use this key else we could use the default operation key (without kind hence will fetch spans for all kinds). With this we can create a migration script to create this new key for all operations, then cache can be refilled with operations and its kinds! If getting spans of all kinds is not a concern then we can skip this and embed the span.kind in operationNameIndexKey and when no tags of span.kind are given we can by default add an unspecified tag while reading that implies when no tags will be given only unspecified spans would be fetched. Migration script here would be adding the kinds in same key. @yurishkuro Please see this and give me direction or a more appropriate way!