Disconnect background process work from request trace

[MadLittleMods]

Before #18849, we we're using our own custom LogContextScopeManager which tied the tracing scope to the LoggingContext. Since we created a new BackgroundProcessLoggingContext any time we run_as_background_process(...), the trace for the background work was separate from the trace that kicked of the work as expected (e.g. request trace is separate from the background process we kicked to fetch more messages from the federation).

Since we've now switched to the ContextVarsScopeManager (in #18849), the tracing scope now crosses the LoggingContext boundaries (and thread boundaries) without a problem. This means we end up with request traces that include all of the background work that we've kicked off bloating the trace and making it hard to understand what's going on.

This PR separates the traces again to how things were before. Additionally, things are even better now since I added some cross-link references between the traces to easily be able to jump between.

Follow-up to #18849

---

In the before, you can see that the trace is blown up by the background process (bgproc.qwer).

In the after, we now only have a little cross-link marker span (start_bgproc.qwer) to jump to background process trace.

Before	After

Testing strategy

1. Run a Jaeger instance (https://www.jaegertracing.io/docs/1.6/getting-started/)

   $ docker run -d --name jaeger \
     -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
     -p 5775:5775/udp \
     -p 6831:6831/udp \
     -p 6832:6832/udp \
     -p 5778:5778 \
     -p 16686:16686 \
     -p 14268:14268 \
     -p 9411:9411 \
     jaegertracing/all-in-one:1.59.0

2. Configure Synapse to use tracing:
   homeserver.yaml

   ## Tracing ##
   opentracing:
    enabled: true
    jaeger_config:
      sampler:
        type: const
        param: 1
      logging:
        false

3. Make sure the optional opentracing dependency is installed: poetry install --extras all
4. In the VersionsRestServlet, modify it to kick off a dummy background process (easy to test this way)

   from synapse.metrics.background_process_metrics import run_as_background_process
   
   async def _qwer() -> None:
       await self.clock.sleep(1)
   
   run_as_background_process("qwer", "test_server", _qwer)

5. Run Synapse: poetry run synapse_homeserver --config-path homeserver.yaml
6. Fire off a version requests: curl http://localhost:8008/_matrix/client/versions
7. Visit http://localhost:16686/search to view the traces
   • Select the correct service
   • Look for the VersionsRestServlet operation
   • Press 'Find Traces' button
   • Select the relevant trace
   • Notice how the trace isn't bloated
   • Look for the start_bgproc.qwer span cross-linking to the background process
   • Jump to the other trace using the cross-link reference -> bgproc.qwer

Dev notes

• Switch to OpenTracing's ContextVarsScopeManager #18849

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct (run the linters)

[anoadragon453]

Nice that we're able to jump between the spans! I appreciate the work to make our observability stack nicer to work with.

Some small requests below, but otherwise this LGTM.

[MadLittleMods]

Recorded the flaky tests on #18537 ⏩

❌ TestThreadReceiptsInSyncMSC4102 ->
https://github.com/element-hq/synapse/actions/runs/18021157721/job/51278706782

[MadLittleMods]

Thanks for the review @anoadragon453 🦈