Merge branch 'development' into mr/cmake

CMakeLists.txt

@@ -1,8 +1,5 @@
 cmake_minimum_required(VERSION 3.13)
-
-project (AMReX)
-
-enable_language(C)
+project(AMReX)
 enable_language(CXX)
 enable_language(Fortran)
 

Docs/sphinx_documentation/source/Basics.rst

@@ -1996,10 +1996,10 @@ that is readable and easy to implement. An example is given below:
         Array4<Real> const& src = fab1.array();
         Array4<Real> const& dst = fab2.array();
 
-        for         (int k = lo.z; k < hi.z; ++k) {
-            for     (int j = lo.y; j < hi.y; ++j) {
+        for         (int k = lo.z; k <= hi.z; ++k) {
+            for     (int j = lo.y; j <= hi.y; ++j) {
                 AMREX_PRAGMA_SIMD
-                for (int i = lo.x; i < hi.x; ++i) {
+                for (int i = lo.x; i <= hi.x; ++i) {
                     dst(i,j,k) = 0.5*(src(i,j,k)+src(i+1,j,k));
                 }
             }
@@ -2020,7 +2020,7 @@ and :cpp:`Box::bigend` of ``bx``.  Both functions return a
 The individual components are accessed by using :cpp:`.x`, :cpp:`.y` and
 :cpp:`.z`, as shown in the :cpp:`for` loops. 
 
-`BaseFab::array()` is called to obtain an :cpp:`Array4` object that is
+:cpp:`BaseFab::array()` is called to obtain an :cpp:`Array4` object that is
 designed as an independent, :cpp:`operator()` based accessor to the
 :cpp:`BaseFab` data. :cpp:`Array4` is an AMReX class that contains a
 pointer to the :cpp:`FArrayBox` data and two :cpp:`Dim3` vectors that
@@ -2041,6 +2041,10 @@ macro on loops that are not safe for vectorization will lead to a variety
 of errors, so if unsure about the independence of the iterations of a
 loop, test and verify before adding the macro.
 
+These loops should always use :cpp:`i <= hi.x`, not :cpp:`i < hi.x`, when 
+defining the loop bounds. If not, the highest index cells will be left out
+of the calculation. 
+
 
 Ghost Cells
 ===========

Docs/sphinx_documentation/source/BuildingAMReX.rst

@@ -364,6 +364,8 @@ below.
    +------------------------------+-------------------------------------------------+-------------+-----------------+
    | BLITZ_INSTALL_DIR            |  Path to Blitz installation directory           |             | user-defined    |
    +------------------------------+-------------------------------------------------+-------------+-----------------+
+   | ENABLE_SUNDIALS              |  Enable SUNDIALS3 interfaces                    | OFF         | ON, OFF         |
+   +------------------------------+-------------------------------------------------+-------------+-----------------+
 .. raw:: latex
 
    \end{center}

Docs/sphinx_documentation/source/BuildingAMReX_Chapter.rst

@@ -13,6 +13,10 @@ its own build system and links to AMReX as an external library.  Finally, AMReX
 can also be built with CMake, as detailed in the section on
 :ref:`sec:build:cmake`.
 
+Please note that we fully support AMReX for Linux systems in general and on the
+DOE supercomputers (e.g. Cori, Summit) in particular.  Many of our users do build
+and use AMReX on Macs but we do not have the resources to fully support Mac users.
+
 .. toctree::
    :maxdepth: 2
 

Docs/sphinx_documentation/source/ForkJoin.rst

@@ -0,0 +1,57 @@
+.. role:: cpp(code)
+   :language: c++
+
+.. role:: fortran(code)
+   :language: fortran
+
+Fork-Join
+=========
+
+An AMReX program consists of a set of MPI ranks cooperating together on
+distributed data.
+Typically, all of the ranks in a job compute in a bulk-synchronous,
+data-parallel fashion, where every rank does the same sequence of
+operations, each on different parts of the distributed data.
+
+The AMReX Fork-Join functionality described here allows the user to divide the
+job's MPI ranks into subgroups (i.e. `fork`) and assign each subgroup
+an independent task to compute in parallel with each other.
+After all of the forked child tasks complete, they synchronize
+(i.e. `join`), and the parent task continues execution as before.
+
+.. figure:: figs/fork_join_tasks.png
+   :scale: 80 %
+   :align: center
+   :alt: Fork-Join Tasks
+
+   Example of a fork-join operation where the parent task's MPI processes (ranks) are
+   split into two independent child tasks that execute in parallel and
+   then join to resume execution of the parent task.
+
+The Fork-Join operation can also be invoked in a nested fashion,
+creating a hierarchy of fork-join operations, where each fork further
+subdivides the ranks of a task into child tasks.
+This approach enables heterogeneous computation and reduces the strong
+scaling penalty for operations with less inherent parallelism or with
+large communication overheads.
+
+.. figure:: figs/nested_fork_join_tasks.png
+   :scale: 80 %
+   :align: center
+   :alt: Nested Fork-Join Tasks
+
+   Example of nested fork-join operations where a child task is further
+   split into more subtasks.
+
+The fork-join operation is accomplished by:
+
+  a) redistributing MultiFab data so that **all** of the data in each
+     registered MultiFab is visible to ranks within a subtask, and
+
+  b) dividing the root MPI communicator into sub-communicators so that
+     each subgroup of ranks in a tasks will only synchronize with each
+     other during subtask collectives (e.g. for ``MPI_Allreduce``).
+
+When the program starts, all of the ranks in the MPI communicator are
+in the root task.
+

Docs/sphinx_documentation/source/index.rst

@@ -3,7 +3,7 @@
 Welcome to AMReX's documentation
 ================================
 
-AMReX is a software framework library containing all the functionality to write
+AMReX is a software framework containing all the functionality to write
 massively parallel, block-structured adaptive mesh refinement (AMR)
 applications. AMReX is freely available `on Github
 <https://github.com/AMReX-Codes/amrex>`_.
@@ -42,6 +42,7 @@ Documentation on migration from BoxLib is available in the AMReX repository at D
    AmrCore_Chapter
    AmrLevel_Chapter
    AsyncIter_Chapter
+   ForkJoin
    IO_Chapter
    LinearSolvers_Chapter
    Particle_Chapter