Describe the bug
While it may not be ideal, i have two distributed instances of Tempo running in the same Namespace
Both are separated clearly by its own labels and its own gossip-ring service .
While everything was running fine for almost an year , suddenly i noticed Traces were getting created in both stores.
Even it was found in S3. Metrics were showing cross-tenant traces as well.

Memberlist was expected to working fine with this. However it didn't.

  join_members:
      - dns+mytempo-prod-gossip-ring:7946

Based on some events thrown by the members, A workaround was attempted to include this param

cluster_label: tempo-prod.<mynamespace>
cluster_label_verification_disabled: false

This helped to fix the issue .

Am wondering , why headless service didnt work as expected. Tried to explore the memberlist module , but haven't got a clarity yet.

To Reproduce
Deploy with in namespace , Check the member list via API query .

Expected behavior
Ideally only own headless service associated members only expected

Environment:
k8s. helm chart - tempo-distributed-1.7.3
Image: tempo:2.3.1-amd64

Additional Context