stabilize worker_total_busy_duration #6899

Owen-CH-Leung · 2024-10-11T04:54:29Z

Motivation

Stabilize worker_total_busy_duration so it can be used outside of --cfg tokio_unstable

Solution

Move the API impl out of tokio_unstable flag

Ref: #6546

…ch and Histogram out of unstable flag

…o WorkerMetrics

rcoh · 2024-10-11T13:30:36Z

tokio/src/runtime/metrics/histogram.rs

+/// Whether the histogram used to aggregate a metric uses a linear or
+/// logarithmic scale.
+#[derive(Debug, Copy, Clone, Eq, PartialEq)]
+#[non_exhaustive]
+#[allow(unreachable_pub)]
+pub enum HistogramScale {
+    /// Linear bucket scale
+    Linear,
+
+    /// Logarithmic bucket scale
+    #[allow(dead_code)]


is this change required for the mean time stabilization or just a coincidental change?

I believe this change is required.

At master we have implemented another struct WorkerMetrics at tokio/src/runtime/metrics/mock.rs which didn't use struct Histogram defined at tokio/src/runtime/metrics/histogram.rs.

In order to stabilize the API, I have to removed this mock WorkerMetrics and instead point to the "real" WorkerMetrics at tokio/src/runtime/metrics/worker.rs

The "real" WorkerMetrics has a field poll_count_histogram which is Option<Histogram>, and thus will attempt to parse tokio/src/runtime/metrics/histogram.rs. From there, you can see Histogram and HistogramBuilder both refers to HistogramScale which is hidden inside tokio_unstable. I think this wouldn't compile.

(I might be wrong though so feel free to correct me)

I need to read through this PR in more detail so bare with me, but we shouldn't be making this stable and public if it isn't actually needed to to stabilize worker_total_busy_duration (and I'm mostly sure that it isn't).

In general, all the #[allow(dead_code)] that are required for this chnage give me the impression that we're exposing something that is only really used in tokio_unstable and so we should find a way to expose it only unstable tokio_unstable.

Thanks for your comment!

I think the core issue here is that by exposing worker_total_busy_duration, we are also exposing the "real" WorkerMetrics at tokio/src/runtime/metrics/worker.rs, which in turn will attempt to parse HistogramScale.

Currently at master branch, when no tokio_unstable flag is passed in, the WorkerMetrics at tokio/src/runtime/metrics/mock.rs will be parsed (which is just an empty struct). And if the flag is passed in, the WorkerMetrics at tokio/src/runtime/metrics/worker.rs will be parsed

https://github.com/tokio-rs/tokio/blob/master/tokio/src/runtime/metrics/mod.rs#L27-L40

it's not actually public, it's still behind config_unstable. (Just verified via the autogenerated docs)

Yeah becoz HistogramScale is still behind cfg_unstable_metrics:

https://github.com/tokio-rs/tokio/pull/6899/files#diff-15ebfca6124144b3cffd9845fa3ce5596c342ce94d84dd406ee84f856db4c66cR23-R25

OK, now I believe I understand.

I would suggest that instead of compiling with the entire worker metrics in order to access only the busy_duration_total field, we should gate all the fields that we won't be using on cfg_unstable_metrics. Otherwise users will still be paying the price for metrics which they don't have access to - and that is something that we would really like to avoid.

Have a look at the runtime Builder for an example of how we have fields and implementations that touch them gated on a cfg flag:

tokio/tokio/src/runtime/builder.rs

Lines 125 to 126 in 512e9de

#[cfg(tokio_unstable)]

pub(super) unhandled_panic: UnhandledPanic,

As a general rule, if you need #[allow(dead_code)] then there is probably something that should be gated on a cfg flag instead.

Thanks @hds . I've reverted changes to histogram.rs. Also the real Histogram, HistogramBuilder and HistogramBatch are gated behind unstable flag now, and instead I've created a mocked version of these for compilation.

For WorkerMetrics, as we target to stabilize more metrics, I'd suggest exposing all fields instead of stabilizing only busy_duration_total and putting other fields behind unstable.

Let me know what you think

@Owen-CH-Leung the problem with stabilizing all of WorkerMetrics is that when tokio_unstable is not enabled, the runtime will pay the price for all those metrics, but there will be no way to access them. For this reason I think that it would be better to only stabilize what we're exposing.

rcoh · 2024-10-11T13:31:15Z

tokio/src/runtime/metrics/runtime.rs

+        pub fn worker_total_busy_duration(&self, worker: usize) -> Duration {
+        let nanos = self
+            .handle
+            .inner
+            .worker_metrics(worker)
+            .busy_duration_total
+            .load(Relaxed);
+        Duration::from_nanos(nanos)
+        }


formatting issues in here—autoformat doesn't run because of the parent macro

Thanks. Fixed the formatting

rcoh · 2024-10-11T13:31:35Z

tokio/src/runtime/scheduler/mod.rs

@@ -9,6 +9,8 @@ cfg_rt! {
    pub(crate) use inject::Inject;

    use crate::runtime::TaskHooks;
+
+    use crate::runtime::{WorkerMetrics};


Suggested change

use crate::runtime::{WorkerMetrics};

use crate::runtime::WorkerMetrics;

Thanks & fixed.

rcoh · 2024-10-11T13:31:47Z

tokio/src/runtime/scheduler/mod.rs

    }

    cfg_unstable_metrics! {
-        use crate::runtime::{SchedulerMetrics, WorkerMetrics};
+        use crate::runtime::{SchedulerMetrics};


Suggested change

use crate::runtime::{SchedulerMetrics};

use crate::runtime::SchedulerMetrics;

Thanks & fixed.

rcoh · 2024-10-11T13:31:57Z

tokio/src/runtime/scheduler/multi_thread/handle/metrics.rs


 cfg_unstable_metrics! {
-    use crate::runtime::{SchedulerMetrics, WorkerMetrics};
+    use crate::runtime::{SchedulerMetrics};


Suggested change

use crate::runtime::{SchedulerMetrics};

use crate::runtime::SchedulerMetrics;

Thanks & fixed.

hds

Please bare with me as I haven't finished the review, but my first impression is that this PR is making more things public than is strictly necessary for the stabilization of worker_total_busy_duration and we should try to avoid that.

This is especially true since #6897 is also touching the histogram (although not taking it out of tokio_unstable) and that would be a breaking change if this PR is released first.

hds · 2024-10-12T16:52:48Z

tokio/src/runtime/metrics/histogram.rs

+/// Whether the histogram used to aggregate a metric uses a linear or
+/// logarithmic scale.
+#[derive(Debug, Copy, Clone, Eq, PartialEq)]
+#[non_exhaustive]
+#[allow(unreachable_pub)]
+pub enum HistogramScale {
+    /// Linear bucket scale
+    Linear,
+
+    /// Logarithmic bucket scale
+    #[allow(dead_code)]


I need to read through this PR in more detail so bare with me, but we shouldn't be making this stable and public if it isn't actually needed to to stabilize worker_total_busy_duration (and I'm mostly sure that it isn't).

In general, all the #[allow(dead_code)] that are required for this chnage give me the impression that we're exposing something that is only really used in tokio_unstable and so we should find a way to expose it only unstable tokio_unstable.

…nges on histogram.rs and guard it behind unstable flag

…on_total, queue_depth and thread_id

stabilize worker total busy duration, bring WorkerMetrics, MetricsBat…

8f1fcb4

…ch and Histogram out of unstable flag

github-actions bot added R-loom-current-thread Run loom current-thread tests on this PR R-loom-multi-thread Run loom multi-thread tests on this PR R-loom-multi-thread-alt Run loom multi-thread alt tests on this PR labels Oct 11, 2024

Fix rustfmt ci job

9b47cf9

Darksonn requested a review from hds October 11, 2024 07:45

Darksonn added A-tokio Area: The main tokio crate M-metrics Module: tokio/runtime/metrics labels Oct 11, 2024

Owen-CH-Leung added 2 commits October 11, 2024 15:52

Fix various failing CI jobs by adding cfg(target_has_atomic = "64") t…

e2f4f33

…o WorkerMetrics

Use cfg_64bit_metrics instead

b6974ba

rcoh reviewed Oct 11, 2024

View reviewed changes

Owen-CH-Leung changed the title ~~stabilize worker total busy duration, bring WorkerMetrics, MetricsBat…~~ stabilize worker_total_busy_duration Oct 11, 2024

Fix formatting and remove brackets

86f019f

Owen-CH-Leung marked this pull request as ready for review October 11, 2024 15:33

hds requested changes Oct 12, 2024

View reviewed changes

Owen-CH-Leung added 2 commits October 16, 2024 22:34

Creat Mock Histogram, HistogramBatch and HistogramBuilder. Revert cha…

64f626d

…nges on histogram.rs and guard it behind unstable flag

Hide queue_depth and thread_id behind unstable flag

489003c

Darksonn mentioned this pull request Oct 16, 2024

Flaky test: single_ack_eliciting_packet_triggers_ack_after_delay quinn-rs/quinn#2014

Open

Owen-CH-Leung added 3 commits October 19, 2024 12:00

Mark most fields of WorkerMetrics as unstable, except for busy_durati…

8a134a2

…on_total, queue_depth and thread_id

Merge branch 'master' into stabilize_worker_total_busy_duration

4bc00cf

Remove allow dead_code, merge master & fix spellcheck

7eb6b97

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stabilize worker_total_busy_duration #6899

stabilize worker_total_busy_duration #6899

Owen-CH-Leung commented Oct 11, 2024

rcoh Oct 11, 2024

Owen-CH-Leung Oct 11, 2024 •

edited

Loading

hds Oct 12, 2024

Owen-CH-Leung Oct 13, 2024

rcoh Oct 14, 2024

Owen-CH-Leung Oct 14, 2024

hds Oct 15, 2024

Owen-CH-Leung Oct 16, 2024

hds Oct 16, 2024

rcoh Oct 11, 2024

Owen-CH-Leung Oct 11, 2024

rcoh Oct 11, 2024

Owen-CH-Leung Oct 11, 2024

rcoh Oct 11, 2024

Owen-CH-Leung Oct 11, 2024

rcoh Oct 11, 2024

Owen-CH-Leung Oct 11, 2024

hds left a comment

hds Oct 12, 2024

	#[cfg(tokio_unstable)]
	pub(super) unhandled_panic: UnhandledPanic,

	use crate::runtime::{WorkerMetrics};
	use crate::runtime::WorkerMetrics;

	use crate::runtime::{SchedulerMetrics};
	use crate::runtime::SchedulerMetrics;

stabilize worker_total_busy_duration #6899

Are you sure you want to change the base?

stabilize worker_total_busy_duration #6899

Conversation

Owen-CH-Leung commented Oct 11, 2024

Motivation

Solution

Choose a reason for hiding this comment

Owen-CH-Leung Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hds left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Owen-CH-Leung Oct 11, 2024 •

edited

Loading