Skip to content

Conversation

@robsyme
Copy link
Collaborator

@robsyme robsyme commented Jan 3, 2026

Summary

Adds a new onWorkflowPublish event to TraceObserverV2 that fires for each value as it is published to a workflow output.

This addresses the request in #6680 for richer context during file publishing. The new event provides:

  • name - the workflow output name (e.g., "alignedBams")
  • value - the complete channel value being published (which may contain files alongside metadata like sample IDs)

Event timeline

Channel value 1 arrives → onWorkflowPublish ← NEW (fires per value)
→ onFilePublish (for each file)
Channel value 2 arrives → onWorkflowPublish ← NEW
→ onFilePublish (for each file)
...
All values complete → onWorkflowOutput (fires once at end)

Unlike onWorkflowOutput which fires after all values have been published, onWorkflowPublish fires in real-time as each value flows through, giving

  1. access to the published values as items in a channel, preserving the channel structure (items in the same channel element are together), and
  2. observers immediate access to the complete channel emission.

Test plan

  • Existing OutputDslTest tests updated to verify notifyWorkflowPublish is called
  • Smoke tests pass

@netlify
Copy link

netlify bot commented Jan 3, 2026

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit 76f2da5
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/695d3bfae4c28b0008ce0105
😎 Deploy Preview https://deploy-preview-6701--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@robsyme robsyme force-pushed the feature-rich-workflow-publish-event branch from 3a8a2d1 to b5a155d Compare January 3, 2026 23:39
Fire a new event during PublishOp::onNext for each value being
published to a workflow output. Unlike onWorkflowOutput which fires
after completion, this provides real-time notification with the
complete channel value as each emission is published.

Signed-off-by: Rob Syme <rob.syme@gmail.com>
@robsyme robsyme force-pushed the feature-rich-workflow-publish-event branch from b5a155d to 6333b8c Compare January 3, 2026 23:44
….groovy

Signed-off-by: Rob Syme <rob.syme@gmail.com>
@bentsherman bentsherman self-requested a review January 5, 2026 20:26
@bentsherman
Copy link
Member

I think this is the right idea

I'm wondering whether we should only fire this event for queue channels and not value channels. Since a value channel is just a single value, the onWorkflowPublish and onWorkflowOutput are identical, and if you use both events, you'll have to be careful to not do something twice. But depending on the use case, it may not matter

Copy link
Member

@pditommaso pditommaso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as a developer I wonder what's the difference of onWorkflowOutput vs onWorkflowPublish vs onFilePublish.

At least there should be a very detailed docs and javadoc describing what event is meant for and when to use it

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman bentsherman linked an issue Jan 6, 2026 that may be closed by this pull request
@bentsherman bentsherman dismissed pditommaso’s stale review January 6, 2026 21:26

Comments were addressed

@bentsherman bentsherman merged commit ffee8a0 into master Jan 6, 2026
24 checks passed
@bentsherman bentsherman deleted the feature-rich-workflow-publish-event branch January 6, 2026 21:26
@bentsherman
Copy link
Member

I updated the java docs to be more comprehensive. I will follow up with a guide in the docs for how to use the trace observer

@robsyme
Copy link
Collaborator Author

robsyme commented Jan 6, 2026

Amazing. Thanks team ❤️ 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enrich FilePublishEvent with source context

4 participants