Skip to content

add configurable polling interval for workflow runs#3397

Open
abelanger5 wants to merge 4 commits intomainfrom
belanger/configurable-wr-polling
Open

add configurable polling interval for workflow runs#3397
abelanger5 wants to merge 4 commits intomainfrom
belanger/configurable-wr-polling

Conversation

@abelanger5
Copy link
Copy Markdown
Contributor

Description

Adds two environment variables to make polling intervals configurable for workflow run subscriptions.

Type of change

  • New feature (non-breaking change which adds functionality)

Copilot AI review requested due to automatic review settings March 25, 2026 12:36
@vercel
Copy link
Copy Markdown

vercel bot commented Mar 25, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hatchet-docs Ready Ready Preview, Comment Mar 25, 2026 8:21pm

Request Review

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 25, 2026

Benchmark results

goos: linux
goarch: amd64
pkg: github.com/hatchet-dev/hatchet/internal/services/dispatcher
cpu: AMD Ryzen 9 7950X3D 16-Core Processor          
                  │ /tmp/old.txt │         /tmp/new.txt          │
                  │    sec/op    │    sec/op     vs base         │
LockAcquisition-8   690.0n ± 68%   694.6n ± 67%  ~ (p=0.937 n=6)

                  │ /tmp/old.txt │         /tmp/new.txt          │
                  │     B/op     │     B/op      vs base         │
LockAcquisition-8   330.0 ± 180%   333.5 ± 173%  ~ (p=0.848 n=6)

                  │ /tmp/old.txt │         /tmp/new.txt         │
                  │  allocs/op   │  allocs/op   vs base         │
LockAcquisition-8    4.000 ± 50%   4.000 ± 50%  ~ (p=1.000 n=6)

Compared against main (7bfc789)

@promptless-for-oss
Copy link
Copy Markdown

📝 Documentation updates detected!

New suggestion: Document workflow run polling interval configuration options


Tip: See how your feedback shapes Promptless in Agent Knowledge Base 🧠

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds configurability for the dispatcher’s workflow-run subscription polling cadence by introducing min/max polling interval settings that can be set via environment variables and passed into the dispatcher.

Changes:

  • Add WorkflowRunPollingMinInterval / WorkflowRunPollingMaxInterval to server runtime config and bind them to new env vars.
  • Thread the configured intervals through engine startup into the dispatcher via new WithMin/WithMaxWorkflowRunPollingInterval options.
  • Switch workflow-run subscription polling from a fixed 1s ticker to a randomized ticker between the configured min/max.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
pkg/config/server/server.go Adds runtime config fields + env bindings for workflow-run polling intervals.
internal/services/dispatcher/server_v1.go Uses a randomized ticker for workflow-run polling in subscription handling.
internal/services/dispatcher/dispatcher.go Adds dispatcher options/fields/defaults for min/max polling intervals.
cmd/hatchet-engine/engine/run.go Wires configured polling intervals into dispatcher initialization.
Comments suppressed due to low confidence (1)

internal/services/dispatcher/server_v1.go:490

  • The random ticker started here is never stopped when the subscription context is canceled. randomticker.NewRandomTicker runs its own goroutine until Stop() is called, so returning on ctx.Done() will leak a goroutine/timer per subscription. Call ticker.Stop() on exit (e.g., defer ticker.Stop() right after creation, or explicitly in the ctx.Done() case).
	go func() {
		ticker := randomticker.NewRandomTicker(s.minWorkflowRunPollingInterval, s.maxWorkflowRunPollingInterval)

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				workflowRunIds := acks.getNonAckdWorkflowRuns()

@@ -479,7 +480,7 @@ func (s *DispatcherImpl) subscribeToWorkflowRunsV1(server contracts.Dispatcher_S

// new goroutine to poll every second for finished workflow runs which are not ackd
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment says this goroutine "poll[s] every second", but the implementation now uses a randomized interval between minWorkflowRunPollingInterval and maxWorkflowRunPollingInterval (and it’s configurable). Update the comment to match the new behavior so future readers don’t assume a fixed 1s poll cadence.

Suggested change
// new goroutine to poll every second for finished workflow runs which are not ackd
// new goroutine to poll at randomized intervals between minWorkflowRunPollingInterval and
// maxWorkflowRunPollingInterval for finished workflow runs which are not ackd

Copilot uses AI. Check for mistakes.
Comment on lines +240 to +250
func WithMinWorkflowRunPollingInterval(interval time.Duration) DispatcherOpt {
return func(opts *DispatcherOpts) {
opts.minWorkflowRunPollingInterval = interval
}
}

func WithMaxWorkflowRunPollingInterval(interval time.Duration) DispatcherOpt {
return func(opts *DispatcherOpts) {
opts.maxWorkflowRunPollingInterval = interval
}
}
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These intervals are user-configurable (via config/env), but randomticker.NewRandomTicker will panic if max <= min (it calls rand.Int63n(max-min)). Consider validating/clamping in New (or in these option setters): require min > 0, max > 0, and max > min (or swap/clamp and log) to prevent a misconfiguration from crashing the dispatcher at runtime.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants