Skip to content

Conversation

@Quinn-With-Two-Ns
Copy link
Contributor

@Quinn-With-Two-Ns Quinn-With-Two-Ns commented Sep 24, 2025

Align how workflow_task_schedule_to_start_latency is measured across the Async and Multithreaded Poller

ProtobufTimeUtils.toM3Duration(response.getStartedTime(), response.getScheduledTime()));

Now both pollers will use the time from the task to calculate workflow_task_schedule_to_start_latency, this also aligns with Core and Go


Note

Compute schedule-to-start latency for async workflow and activity pollers using task-provided timestamps instead of local time.

  • Metrics:
    • Workflow poller: WORKFLOW_TASK_SCHEDULE_TO_START_LATENCY now computed from task timestamps (r.getStartedTime()r.getScheduledTime()).
    • Activity poller: ACTIVITY_SCHEDULE_TO_START_LATENCY now computed from task timestamps (r.getStartedTime()r.getCurrentAttemptScheduledTime()).

Written by Cursor Bugbot for commit 402f814. This will update automatically on new commits. Configure here.

@Quinn-With-Two-Ns Quinn-With-Two-Ns marked this pull request as ready for review September 24, 2025 23:35
@Quinn-With-Two-Ns Quinn-With-Two-Ns requested a review from a team as a code owner September 24, 2025 23:35
Comment on lines -160 to +158
.record(ProtobufTimeUtils.toM3Duration(startedTime, r.getScheduledTime()));
.record(ProtobufTimeUtils.toM3Duration(r.getStartedTime(), r.getScheduledTime()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any idea of what the practical time difference could be for users that may be monitoring this metric? I guess we can just call it out in release notes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clock skew between the Server and Worker, maybe some unaccounted for network time. The main issue is the async poller was measuring it differently then everywhere else which is a bug.

Comment on lines -160 to +158
.record(ProtobufTimeUtils.toM3Duration(startedTime, r.getScheduledTime()));
.record(ProtobufTimeUtils.toM3Duration(r.getStartedTime(), r.getScheduledTime()));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see this same logic for activity, do we want to update that one to use the activity poll response start time too? I also saw this logic with Nexus, but no start time on its response (I also confirmed the activity and workflow started time fields on the response go as far back as Temporal, so we're safe for old server versions)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah good catch, fixed

cursor[bot]

This comment was marked as outdated.

@Quinn-With-Two-Ns Quinn-With-Two-Ns force-pushed the align_wft_schedule_to_start_latency branch 2 times, most recently from b84aa7a to 647f690 Compare September 26, 2025 17:12
@Quinn-With-Two-Ns Quinn-With-Two-Ns force-pushed the align_wft_schedule_to_start_latency branch from 647f690 to 402f814 Compare October 22, 2025 19:03
@Quinn-With-Two-Ns Quinn-With-Two-Ns merged commit 850515b into temporalio:master Oct 22, 2025
24 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants