Wishlist from a targets user

[nviets]

I'm a regular targets user and excited to see @wlandau's use of Mirai in crew/targets. As a longtime targets user, I'm used to the clustermq backend, and there are a few requests that came to mind after reading some of the recent Mirai threads. Here they are:

1. Heterogeneous workers - In complex DAGs, it would be nice to tag targets for specific worker configurations. Some use cases include executing lightweight tasks on tiny workers and farming heavy modeling tasks to large workers and specialized workers for hardware specific operations, e.g. a GPU worker on the specific node. I recall clustermq had limitations on mixed worker configurations.
2. Arrow flight support - Targets already supports the arrow format for caching objects, but can it send data between targets (or from one to many in branching operations) via Arrow Flight? Caching intermediate operations to disk becomes unnecessary if data can be shared in a high performance way over the network.
3. Programmable port ranges - In an enterprise context, it's often necessary to work within restricted port ranges. Will targets/crew/mirai expose settings for this config?

[wlandau]

Hi @nviets,

1. Whether through rrq/Redis or through mirai, I plan for crew to handle heterogeneous workers. In mirai, Multiple client sockets #23 will allow crew to maintain a small number of worker groups, where workers are homogeneous within group and heterogeneous from group to group.
2. I don't know enough to answer this one, but it sounds interesting and potentially helpful.
3. @shikokuchuo made this possible in mirai, and crew will have options to specify the port you use. As you may have seen in the discussion threads, the mirai active queue needs one port per worker, which is a challenge for me, so I am building on the passive queue for the moment.

[shikokuchuo]

Just a comment on 2. Arrow flight. Messaging via 'mirai' goes via R serialisation to a binary transfer directly vs. translation into Arrow format & packing to protobuf. I am sure Arrow has improved in the meantime but previously it did not capture all data attributes on R objects correctly.

[nviets]

Thanks! Sounds like 1 and 3 are covered. Regarding 2, arrow only covers a small but important subset of R data objects, basically tabular data structures. In targets today, I see arrow is already used to cache and retrieve data on disk, but arrow/flight also optimizes over the network transfers of data between separate R instances (and operations, think dplyr, on the data in motion) and exchanges between languages, python, julia, C++, etc... In cases where one target passes tabular data with one or more downstream targets, is the transfer best handled by arrow or by nanonext?

[wlandau]

c.f. wlandau/crew#23 (reply in thread)

[shikokuchuo]

Thanks @nviets for your comments and ideas. As https://github.com/wlandau/crew is now fully-realised and integrated into https://github.com/ropensci/targets and the various {crew} plugins start to come online such as https://github.com/wlandau/crew.cluster do feel free to post further suggestions.

[nviets]

Thanks @shikokuchuo and @wlandau! I am really excited to try your latest work. I'll update soon and share some feedback on the experience relative to clustermq. On my end, it takes a bit of retooling internal libraries against your new packages, so there's a bit of a delay from when you release to CRAN. Congratulations on your awesome collaboration here! :)