wrong tracing spans hierarchy in `mergeQuerier` meethods

[vlad-diachenko]

Describe the bug
mergeQuerier uses a list of queriers under hood and if we call any method of mergeQuerier, for example Select(), it executes Select method of some(or all) queriers from this list.
In this PR was added tracing spans to some methods of mergeQuerier :

• mergeQuerier.labelValues
• mergeQuerier.labelNames
• mergeQuerier.select

So, If we start tracing span in some method of mergeQuerier then all tracing spans started during this method execution must be marked as children of span from mergeQuerier.
However, it is not true for current implementation.

AR:
If we take a look to tracing visualisation, we can see that spans in greed rectangle are children of span promqlPrepare even if these methods was executed from mergeQuerier.Select method.

This issue is reproduced for all spans started in mergeQuerier.

To Reproduce
Steps to reproduce the behavior:

1. update config to enable auth and tenant_federation :

auth_enabled: true
tenant_federation:
  enabled: true

2. Start Cortex (#4147 added tenant_id tag to tracing spans #4186). for example: using docker-compose file development/tsdb-blocks-storage-s3-single-binary/docker-compose.yml
3. Perform Operations(Read) with a few tenants defined in X-Scope-OrgID header. for example:

curl --location --request GET 'http://localhost:8001/prometheus/api/v1/query_range?query=up&start=2021-05-12T06:45:00Z&end=2021-05-12T15:01:00Z&step=10m' \
--header 'X-Scope-OrgID: team-a|team-b'

Expected behavior
The span started in querier from inner list of mergeQuerier, must be a child of span started in mergeQuerier.

Environment:

• Infrastructure: laptop
• Deployment tool: docker-compose file development/tsdb-blocks-storage-s3-single-binary/docker-compose.yml

Storage Engine

• Blocks
• Chunks

Additional Context

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.