Skip to content

Commit

Permalink
Merge pull request bsc-wdc#347 from bsc-wdc/ds_array_documentation
Browse files Browse the repository at this point in the history
ds-array documentation - irregular blocks
  • Loading branch information
michal-choinski committed Sep 8, 2021
2 parents bb6303d + 310d56a commit 6f86937
Showing 1 changed file with 18 additions and 2 deletions.
20 changes: 18 additions & 2 deletions docs/source/user-guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,8 @@ All operations on ds-arrays are internally parallelized with PyCOMPSs. The
degree of parallelization is controlled using the array's block size. Block
size defines the number of rows and columns of each block in a ds-array.
Sometimes a ds-array cannot be completely split into uniform blocks of a
given size. In these cases, some blocks of the ds-array will be slightly
smaller than the defined block size. Choosing the right block size is essential
given size. In these cases, the bottom and right blocks will be smaller than
the specified regular block size. Choosing the right block size is essential
to be able to exploit dislib's full potential.

Choosing the right block size
Expand Down Expand Up @@ -175,6 +175,22 @@ Nevertheless, these are the main ideas when choosing your block size:
3. For small ds-arrays, it might be better to use N < number of processors
and increase granularity at the cost of reducing parallelism.

Irregular blocks
................

Usually a ds-array cannot be divided into blocks of equal size. In this case,
bottom-most blocks can be shorter than the regular block size, and right-most blocks
can be narrower. When both dimension are not divisible by the desired block height/weight,
bottom-right block is both shorter and narrower.

However, there is also a special case for sliced ds-arrays. If the performed slicing begins
in middle of the block, this block is sliced and its new dimensions are kept as
the top-left block instead of re-chunking the whole array. The resulting ds-array would then have the top
and the left blocks smaller than the regular size blocks. Furthermore, it could also have
the bottom and the right blocks smaller as well, depending on the original shape of the matrix,
and the bottom/right bounds used in slicing. This optimization helps to avoid unnecessary
and costly re-chunking of the ds-array.

Creating arrays
...............

Expand Down

0 comments on commit 6f86937

Please sign in to comment.