Issues with padding and the irregular block distribution

[bgrant]

The width of communication padding must not exceed the number of indices owned by the neighbor indicated. That seems clear.

However, given this constraint and the constraint that all communication padding in a dimension must be equal, this means that (in an irregular-block distribution) the amount of communication padding is constrained by the local array with the smallest number of indices.

For example, in a four-process 1-d grid like so:

[10 elements] [10 elements] [0 elements] [0 elements]

no communication padding would be allowed. Additionally, if right-side boundary padding is specified, currently this would be forced to exist on process 3 (in other words, on an empty process, not proximate to data).

[wfspotz]

Interesting corner case. You would certainly want to allow communication between processors 0 and 1 via communication padding, but there is no communication possible between processors 1 and 2 or between 2 and 3. One way you might come up with this arrangement is through slicing. However, in my implementation (Domi), I obtain a new sub communicator when I slice, giving me only processors 0 and 1 in this case.

If you are going to allow a distribution as you propose in your example, then non-constant communication padding sizes will need to be allowable. The minimal constraint would be that the lower padding on processor p+1 would need to equal the upper padding on processor p.

[bgrant]

What do you think about the boundary padding being attached to the first and last processes in the grid, even if there isn't data on those processes?

[wfspotz]

Another corner case I prevent in Domi. If the number of elements on a processor is zero, then I'm OK with requiring that the boundary and communication padding be zero also.

[kwmsmith]

That makes sense to me for communication padding, since it is intended to be used for communication with the neighboring process. But it seems inconsistent with boundary padding: it shouldn't care about what's going on in the neighboring process, since boundary padding is not an artifact of the array's distribution.

For instance, if someone redistributes a nice regular block array with boundary and communication padding into an irregular distribution that has empty processes, then boundary padding should remain unchanged, while communication padding should adjust as required.

[wfspotz]

I though of another use case: sending an entire distributed array to a single processor. This is often not possible due to memory constraints, but a lot of times it is possible, and people like to do it so that they can use, say, a post-processing tool that is not parallelized, especially when debugging a small problem.

In this case, the entire array, boundary padding in tact, would be sent to, say, processor zero (or the interactive ipython processor), and the other processors would would have zero size. In Domi, I would do this as a sub-communicator, but you guys would need to figure out how this common use case would be handled.

[kwmsmith]

@wfspotz -- regarding sending all data to a single processor: we have a toarray() (and tondarray() alias) in DistArray to handle this case. Like you say, sometimes this isn't possible depending on the size of the result, but when it is possible, it is really handy for testing, visualization, etc.

For the other case, when you want to bring the entire array onto a single rank / engine (not the IPython client process), we'd use a redistribution method to do that. And yes, we'd make sure to include boundary padding as part of the result.

[kwmsmith]

Addressed by #97