Skip to content

Mini Julep: Data Parallelism Low Hanging Fruit #36579

Open

Description

Why?

Data-parallel functions like pmap and an a hypothetical tmap are not answers to everything, but they are answers to a lot of things. If we could at least support these kinds of primitives everywhere, including source-to-source AD like Zygote, then the parallelism needs of the 90% are solved and we can figure out later how to do the last 10%.

What?

There are a few things to do:

  • Agree on how to support tmap, and possibly ptmap, i.e. EnsembleSplitThreads of DiffEq where it mixes multithreading on each process.
  • Make it easier to put one process per node, i.e. a smart addprocs
  • Have all of the ecosystem like Zygote support the whole set of primitives. This has started in support parallel maps (tmap, pmap, vmap, etc.) FluxML/Zygote.jl#728 .

Maybe if they could be linked better this could be easier? Like map(f,Parallel(),args...) or something, but it's probably too late for that so the low hanging fruit is just to do a different function for each.

For tmap, there's currently:

but both are not registered. There's nothing for ptmap.

@tkf might have some ideas.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    julepJulia Enhancement Proposal

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions