Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add waitany and waitall functions to wait multiple tasks at once #53341

Merged
merged 30 commits into from
Mar 11, 2024

Conversation

mrkn
Copy link
Contributor

@mrkn mrkn commented Feb 15, 2024

I would like to propose adding two functions, waitany and waitall, discussed in the issue #53226. These functions wait for multiple tasks at once. The waitany function blocks until one task finishes. The waitall function blocks until all tasks finish. There is an optional keyword argument, failfast, for the waitall function. The default of failfast is false. The waitall function will immediately stop if any task ends with an exception when the failfast is true.

This is my own implementation, but I have regrets about the type of the first argument. I wanted to represent a container type from which Task objects can be taken out using the iterate function, but it seems impossible in current Julia, so I used a union type of AbstractVector, Tuple, and Set. I would like to know if there is a better way to write this part.

@oscardssmith oscardssmith added the multithreading Base.Threads and related functionality label Feb 15, 2024
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
@mrkn mrkn requested a review from vtjnash March 1, 2024 09:22
base/task.jl Outdated Show resolved Hide resolved
Copy link
Member

@vtjnash vtjnash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems quite great. It has got lots of annoying details if anyone was trying to emulate it themself, but seems fairly straightforward internally once you work out those details. I think we are ready to merge it, unless there is anything else of concern with it from anyone? One observation I might suggest is that we add another flag throw (related to failfast) that defines whether it should fail with CompositeException when exception is set (independent of failfast)?

This may also be a good drop-in replacement for the Experiment.sync_end implementation, with waitall and failfast, which may be a good demo of it in use?

base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
base/task.jl Outdated Show resolved Hide resolved
@mrkn mrkn force-pushed the wait_multiple_tasks branch 4 times, most recently from db5ef50 to dace55a Compare March 6, 2024 15:25
@vtjnash vtjnash merged commit 8413b97 into JuliaLang:master Mar 11, 2024
5 of 7 checks passed
@mrkn mrkn deleted the wait_multiple_tasks branch March 11, 2024 15:36
@quinnj
Copy link
Member

quinnj commented Mar 11, 2024

Thanks for all the persistence and effort here @mrkn! Really great to get this in!

IanButterworth pushed a commit that referenced this pull request Mar 11, 2024
As we discussed with @vtjnash in PR #53341, it might be useful to
introduce the `throw` option in the `wait` function for `Task`. If
`throw=false` is specified, `wait` behaves like `_wait`; it prevents
throwing a `TaskFailedException`.
@IanButterworth IanButterworth added the backport 1.11 Change should be backported to release-1.11 label Jun 7, 2024
@IanButterworth
Copy link
Member

Adding backport to 1.11 because we want this in 1.11 for #54674

@IanButterworth
Copy link
Member

IanButterworth commented Jun 11, 2024

#54760 means we no longer need waitany on 1.11 so I'm removing the backport label

@IanButterworth IanButterworth removed the backport 1.11 Change should be backported to release-1.11 label Jun 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
multithreading Base.Threads and related functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants