Skip to content

Update the halo documentation to cover the new exchange interfaces #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
+66 −3
Draft

Update the halo documentation to cover the new exchange interfaces #8

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 66 additions & 3 deletions docs/source/pages/api_halo.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,18 @@ Here the first parameter var, a 3D input array, contains the normal pencil-distr

As with the rest of the 2DECOMP&FFT library, a more general form of the routine is available (implemented using Fortran optional parameters):

``call update_halo(var, var_halo, level, opt_decomp, opt_global)``
``call update_halo(var, var_halo, level, opt_decomp, opt_global, opt_pencil)``

This supports halo-cell communications among pencils with arbitrary global sizes, as described by `opt_decomp`, the decomposition object. The last optional parameter `opt_global` is required (to be set to .true.) if global coordinate is used to define the pencils, i.e. the input array var is defined using the start/end variables rather than size variables. This ensures the coordinate systems used by `var` and `var_halo` are consistent.
This supports halo-cell communications among pencils with arbitrary global sizes, as described by
`opt_decomp`, the decomposition object.
The optional parameter `opt_global` is required (to be set to .true.) if global coordinate is used
to define the pencils, i.e. the input array var is defined using the start/end variables rather than
size variables.
This ensures the coordinate systems used by `var` and `var_halo` are consistent.
The optional parameter `opt_pencil` allows the user to specify which pencil they are operating in
(`1=X,2=Y,3=Z`), our recommendation is to do so, the current interface makes this parameter optional
to continue supporting the previous implementation which would try to determin the orientation
automatically, however this may not be reliable.

To demonstrate the use of this API, here is an example that computes spatial derivatives:

Expand Down Expand Up @@ -49,4 +58,58 @@ The extra parameter periodic_bc is a 1D array containing 3 logical values that s

Like the rest of 2DECOMP&FFT, the halo-cell support API is implemented in a black-box fashion. The library internally handles the communications between neighbouring blocks using standard MPI non-blocking point-to-point communications.


-----------------------
Halo-exchange interface
-----------------------

The high-level halo interface described above makes it easy to add halo support to existing codes.
However, by handling the memory allocation for you, it is difficult to adopt in a code that is
already written to handle its halos when porting to the 2DECOMP&FFT library.
As described above, the `update_halo` interface operates in two phases:
1) a new array, extended with halo entries is allocated
2) local data is copied into the local region of this array and used to communicate halo values to
neighbouring processes

To facilitate the use of 2DECOMP&FFT by existing stencil codes we have exposed the data exchange
(step 2) as a new interface `exchange_halo`.
There are two behaviours of the interface based on the optional parameters `opt_depth` and
`opt_levels`

::
call halo_exchange(arr, ipencil, opt_depth, opt_levels)

where `arr` is the working array, `ipencil=1,2,3` indicates the pencil orientation and the optional
values `opt_depth` and `opt_levels` take scalar and array integer values to indicate the halo
depths.
The former sets the halo depth in the two perpendicular directions to the current pencil orientation
and the latter allows an arbitrary depth of halo to be set
in each axis of the current orientation.
The working array is expected to be prepopulated with local data and of size
`(nx+2*ih,ny+2*jh,nz+2*kh)`, where `ih,jh,kh` are the halo depths in each direction.
To illustrate the use of `exchange_halo` the two calls

::
call halo_exchange(vh, 1, opt_depth=1)
call halo_exchange(wh, 1, opt_levels=[0, 1, 1])

are equivalent and will exchange halo data to depth 1 in the Y and Z axes for data in the X pencil.

To aid users in creating suitably-sized arrays the allocation subroutines have been extended with
optional halo parameters `opt_depth` and `opt_levels` whose meaning matches the above.
When supplied with non-zero values 2DECOMP&FFT will allocate a halo-padded array to the requested
depth.

When a code is heavily dependent on halo exchanges, it may be repetitive to specify the halo depths
at each exchange.
In such cases, the user can specify a default depth for a given decomposition by setting

::
decomp_info%xlevel = [ihx, jhx, khx]
decomp_info%ylevel = [ihy, jhy, khy]
decomp_info%zlevel = [ihz, jhz, khz]

if `opt_depth/opt_levels` is not specified in the allocation, or halo exchange routines, then this
value will be used (note that by default these are all set to `[0,0,0]`).

We would like to acknowledge the work of S.Owens in the ARCHER2 eCSE05-03 project
(https://www.archer2.ac.uk/ecse/reports/eCSE05-03/) who initially developed these interfaces.
Loading