feat(metrics): Add recurring job label to backup metrics or empty value

[YvanGuidoin]

Which issue(s) this PR fixes:

Issue N/A

What this PR does / why we need it:

Monitoring on usage of Backup per recurring job so we can trace the type of backup applied by PVC

Special notes for your reviewer:

Tried to understand documentation here and on longhorn-tests, but I am thoroughly confused on how to test a simple code change without creating a new version.

make/make test and CI seems to work though.

Additional documentation or context

Our recurring jobs, to trace usage/granularity per PVC, so we can raise alarms on certain pods not being backed up enough etc...

---
apiVersion: longhorn.io/v1beta1
kind: RecurringJob
metadata:
  name: backup-weekly
  namespace: longhorn-system
spec:
  cron: "@weekly"
  task: "backup"
  groups:
    - backup-weekly
    - backup-daily
    - backup-hourly
  retain: 12
---
apiVersion: longhorn.io/v1beta1
kind: RecurringJob
metadata:
  name: backup-daily
  namespace: longhorn-system
spec:
  cron: "@daily"
  task: "backup"
  groups:
    - backup-daily
    - backup-hourly
  retain: 7
---
apiVersion: longhorn.io/v1beta1
kind: RecurringJob
metadata:
  name: backup-hourly
  namespace: longhorn-system
spec:
  cron: "@hourly"
  task: "backup"
  groups:
    - backup-hourly
  retain: 24

[derekbit]

cc @ChanYiLin @c3y1huang for helping review it.

[c3y1huang]

LGTM

[c3y1huang]

We should also update the document.
https://github.com/longhorn/website/blob/master/content/docs/1.8.0/monitoring/metrics.md?plain=1

@YvanGuidoin could you help? Thank you.

[YvanGuidoin]

@c3y1huang I can help end of next week as I am currently travelling

[c3y1huang]

@YvanGuidoin I've created a feature request for this. In the future, I encourage you to create it yourself next time to have it included in the development cycle. Thank you.

longhorn/longhorn#9429

cc @derekbit

[c3y1huang]

cc @ChanYiLin

[ChanYiLin]

Hi @YvanGuidoin
For the testing of this feature, we can rely on @longhorn/qa to verify after the PR is merged
longhorn/longhorn#9429 (comment)
You can elaborate the testing steps and the expected results in this comment so the QA team members can test it on their own environment.

Some enhancement features or function changing features may need to design an e2e test in longhorn-test repo, but there is no need for this one.
Thanks for your contribution.