Expose message-bytes-limit in config
#7074
Conversation
| @@ -339,6 +339,17 @@ properties: | |||
| To avoid this, Dask usually used a file-based lock. | |||
| However, on some systems file-based locks don't work. | |||
| This is particularly common on HPC NFS systems, where users may want to set this to false. | |||
| transfer: | |||
There was a problem hiding this comment.
XREF: #6977
I went with singular transfer to be coherent with variable naming.
| @@ -1129,7 +1130,7 @@ class WorkerState: | |||
| #: :meth:`BaseWorker.gather_dep`. Multiple small tasks that can be fetched from the | |||
| #: same worker will be clustered in a single instruction as long as their combined | |||
| #: size doesn't exceed this value. | |||
| transfer_message_target_bytes: int | |||
| transfer_message_bytes_limit: float | |||
There was a problem hiding this comment.
Driveby: Renamed to match other attribute names (e.g., transfer_incoming_bytes_limit). cc @crusaderky
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 15 files ±0 15 suites ±0 6h 15m 28s ⏱️ -19s For more details on these failures, see this check. Results for commit 71df3cd. ± Comparison against base commit 9038c7a. |
| - string | ||
| - integer | ||
| description: | | ||
| The maximum size of a message sent between workers |
There was a problem hiding this comment.
This description is not correct, no? Rather, IIUC, this is the maximum amount of "extra" data we'll ask for from a worker once we've picked a worker to transfer a task from.
So an individual message may be (much) larger than message-bytes-limit, but if we decide to glue a bunch of messages together into one gather the sum of the size of those messages will never exceed message-bytes-limit.
There was a problem hiding this comment.
It does not explain the fine print, that's a fair point. I'll file a follow-up PR that highlights that this is not an absolute maximum.
|
thank you! |
Addresses
Conflicts with
_select_keys_for_gather#7071 (requires adjustments)pre-commit run --all-files