Add more scheduler metrics

[theAntiYeti]

Added metrics:

• Jobs Considered: A count of the number of jobs considered over the lifetime of the scheduler.
• Fair share per queue
• Actual share per queue

I think the difference from fair share factor could probably just be a derived metric in thanos or somewhere else?

┆Issue is synchronized with this Jira Task by Unito

[severinson]

Suggested change

	func (s *Scheduler) cycle(ctx context.Context, updateAll bool, leaderToken LeaderToken, shouldSchedule bool) (*SchedulerResult, error) {
	var overallSchedulerResult *SchedulerResult
	func (s *Scheduler) cycle(ctx context.Context, updateAll bool, leaderToken LeaderToken, shouldSchedule bool) (overallSchedulerResult *SchedulerResult, err error) {

[severinson]

You can define variables in the return section of the function definition. Then you can just type return later.

[d80tb7]

I don't like returning nil for the schedulerResult here as could lead to panics. I think this might already be the case because it looks to me as if line 183 could pass a nil through to s.metrics.ReportSchedulerResult in the case where we lose leadership midway through the cycle.

My solution would be to return an object that is always no null and contains a bool to say if we scheduled or not and a SchedulerResult that is guaranteed to be non-nil if the bool is true.

[d80tb7]

not sure what err is here but if it's an actual error do something like:

logging.WithStacktrace(log, err).Errorf("error reteriving considered jobs observer for queue %s, pool %s", queue, pool)

[d80tb7]

as above- log error more verbosely

[d80tb7]

could we extract the logic to calculate queue weight somewhere central (the schedContext?) so we don't have this defined in multiple places? @severinson?

[d80tb7]

as above- log error more verbosely