Simplify tracking of task counts stats in TaskQueue

[kfaraz]

Description

The current model of tracking task counts for the TaskCountStatsMonitor is too limiting and verbose.

• It requires a different method for each type of task count - running, waiting, pending, etc.
• It makes adding new dimensions and metrics cumbersome as can be seen in the PR adds taskType dimension to task count metrics #16403

Changes

• Remove deprecated methods from TaskCountStatsProvider
• Rename CoordinatorRunStats to DruidRunStats as it is now being extensively used outside the coordinator.
• Simplify collection of stats in TaskQueue.
  • Remove methods getRunningTaskCount, getWaitingTaskCount, etc.
  • Collect all stats in a single DruidRunStats instance which is reset upon invocation of getQueueStats()
• Add dimension taskType to task count metrics

Classes to review

• TaskQueue
• TaskCountStatsProvider
• TaskCountStatsMonitor
• TaskMaster

Further work

• There are several other count stats providers which may benefit from a similar cleanup. I will try to consolidate all of them in a follow up PR.

Design considerations

The few advantages of the previous design were that:

• The metrics emitted by the monitor were clearly listed out in the monitor itself. This can still be retained by adding logic in the monitor to filter out the metrics contained in the DruidRunStats.
• The individual methods in the interface made it necessary for the concrete implementation to provide those metrics. But with a plain getStats() method which returns a packaged DruidRunStats, it is possible for the concrete implementation to return any or no metrics inside the DruidRunStats. Still thinking of a clean way to enforce this. I suppose we could add logic in the monitor to ensure that all required metrics have been sent by the implementation.

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[github-actions]

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

[github-actions]

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

[github-actions]

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

[kfaraz]

I don't intend to merge this PR as it has too many conflicts and the approach might need to be changed as well.
But keeping it open as there are some changes here that I might be able to salvage.

[github-actions]

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.