metrics, observer: purge backend metrics when backend is down for too long

[djshow832]

What problem does this PR solve?

Issue Number: close #582

Problem Summary:
The backend metric is never cleared after the backend is down. In auto-scaling workload, the backends change frequently and then the metrics keep growing.

What is changed and how it works:

• Purge the backend metrics after it's down for over 2 hours
• Remove the status label for BackendStatusGauge. BackendStatusGauge is not used in Grafana so it's fine.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Steps:

1. Update the threshold from 2 hours to 2 minutes and build TiProxy
2. Create a cluster with tiup playground v8.1.0 --db=2 --tiproxy=1 --tiproxy.version=v1.1.0 --tiflash=0
3. Run sysbench to create metrics
4. Scale-in the TiDB 127.0.0.1:4001
5. Check the metrics immediately and 127.0.0.1:4001 existed

curl -L 127.1:3080/metrics | grep -c 127.0.0.1:4001
204

6. Check the metrics after a few minutes and 127.0.0.1:4001 disappeared:

curl -L 127.1:3080/metrics | grep -c 127.0.0.1:4001
0

7. Check the Grafana, the metrics of 127.0.0.1:4001 existed when it was up but disappeared after it was down:

Notable changes

• Has configuration change
• Has HTTP API interfaces change
• Has tiproxyctl change
• Other user behavior changes

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

- Purge backend metrics when the backend is down for too long

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 93.50649% with 5 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@1ace159). Learn more about missing BASE report.

Files	Patch %	Lines
pkg/balance/observer/backend_observer.go	70.58%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #585   +/-   ##
=======================================
  Coverage        ?   67.85%           
=======================================
  Files           ?       76           
  Lines           ?     6953           
  Branches        ?        0           
=======================================
  Hits            ?     4718           
  Misses          ?     1899           
  Partials        ?      336           

Flag	Coverage Δ
unit	67.85% <93.50%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: xhebox

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [xhebox]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-07-03 02:52:09.788811941 +0000 UTC m=+1379256.274300774: ☑️ agreed by xhebox.