Merge branch 'development'

CHANGES

@@ -0,0 +1,24 @@
+# changes since the last releas
+
+  -- Added a load balance approach to Amr/AmrLevel.  The runtime
+     parameters for this is amr.loadbalance_with_workestimates and
+     amr.loadbalance_level0_int.  The latter is for single level
+     (i.e., amr.max_level=0) only.  For multi-level runs, load balance
+     is done during regrid and thus the load balance interval is
+     controlled by amr.regrid_int.  To use this approach, AmrLevel
+     derivative must has a StateData for work estimates and its index
+     is returned by virtual function int WorkEstType().
+
+  -- MFIter now supports dynamic scheduling of OpenMP threads.  For
+     example,
+
+       for (MFIter mfi(mf, MFItInfo().SetDynamic(true).EnableTiling(tile_size);
+            mfi.isValid(); ++mfi)
+       { ... }
+
+  -- Added EBFluxRegister to Src/EB for refluxing and
+     re-redistribution in EB.
+
+  -- amrex.signal_handling = 1 is a new runtime parameter that can be
+     used to control whether AMReX should handle signals like SIGSEGV
+     etc. 

Docs/AMReXUsersGuide/AMReXUsersGuide.tex

@@ -331,6 +331,10 @@ \chapter{Particles}\label{Chap:Particles}
 \chapter{Fortran Interface}\label{Chap:Fortran}
 \input{Fortran/Fortran}
 
+
+\chapter{Embedded Boundaries}\label{Chap:EB}
+\input{EB/EB}
+
 \chapter{Visualization}\label{Chap:Visualization}
 \input{Visualization/Visualization}
 

Docs/AMReXUsersGuide/CVODE/CVODE.tex

@@ -1,14 +1,17 @@
-\amrex\ supports ODE integration using the \cvode\ solver,\footnote{\url{https://computation.llnl.gov/projects/sundials/cvode}} which is part of the \sundials\ framework.\footnote{\url{https://computation.llnl.gov/projects/sundials}}
+\amrex\ supports local ODE integration using the \cvode\ solver,\footnote{\url{https://computation.llnl.gov/projects/sundials/cvode}} which is part of the \sundials\ framework.\footnote{\url{https://computation.llnl.gov/projects/sundials}}
 \cvode\ contains solvers for stiff and non-stiff ODEs, and as such is well suited for solving e.g., the complex chemistry networks in combustion simulations, or the nuclear reaction networks in astrophysical simulations.
 
 Most of \cvode\ is written in C, but many functions also come with two distinct Fortran interfaces.
 One interface is \fcvode, which is bundled with the stable release of \cvode.
-Its usage is described in the CVODE documentation (\url{https://computation.llnl.gov/sites/default/files/public/cv_guide.pdf}).
+Its usage is described in the \cvode\ documentation.\footnote{\url{https://computation.llnl.gov/sites/default/files/public/cv_guide.pdf}}
+However, the use of \fcvode\ is discouraged in \amrex\ due to its incompatibility with being used inside OpenMP parallel regions (which is the primary use case in \amrex\ applications).
 
-The second, newer Fortran interface to \cvode\ uses the \texttt{iso\_c\_binding} feature of the Fortran 2003 standard to implement a ``thinner,'' more direct interface to the C functions in \cvode.
-In contrast to the \fcvode\ interface, the Fortran 2003 interface calls C functions directly, with identical arguments.
-When compiling \cvode, one need not build the Fortran interface to \cvode\ at all to use this new interface.
-The examples provided in \amrex\ use this new interface.
+The alternative, and recommended, Fortran interface to \cvode\ uses the \texttt{iso\_c\_binding} feature of the Fortran 2003 standard to implement a direct interface to the C functions in \cvode.
+When compiling \cvode, one need not build the \cvode\ library with the \fcvode\ interface enabled at all.
+Rather, the Fortran 2003 interface to \cvode\ is provided within \amrex\ itself.
+The \cvode\ tutorials provided in \amrex\ use this new interface.
+
+\section{Compiling \cvode}
 
 To use \cvode\ in an \amrex\ application, follow these steps:
 
@@ -41,6 +44,32 @@
     /path/to/cvode/or/sundials/top/level/source/dir
   \end{verbatim}
 
+  One can supply additional flags to CMake or to the compiler to customize the compilation process. 
+  Flags of interest may include \texttt{CMAKE\_C\_FLAGS}, which add the specified flags to the compile statement, e.g.,
+  \texttt{-DCMAKE\_C\_FLAGS="-h list=a"} will append the \texttt{-h list=a} flag to the \texttt{cc} statement when compiling the source code.
+  Here one may wish to add something like \texttt{"-O2 -g"} to provide an optimized library that still contains debugging symbols; 
+  if one neglects debugging symbols in the \cvode\ library, and if a code that uses \cvode\ encounters a segmentation fault in the solve, 
+  then the backtrace has no information about where in the solver the error occurred.
+  Also, if one wishes to compile only the solver library itself and not the examples that come with the source 
+  (compiling the examples is enabled by default), one can add \texttt{"-DEXAMPLES\_ENABLE=OFF"}.
+  Users should be aware that the \cvode\ examples are linked dynamically, so when compiling the solver library on Cray system using the Cray compiler wrappers \texttt{cc}, \texttt{CC}, and \texttt{ftn}, one should explicitly disable compiling the examples via the \texttt{"-DEXAMPLES\_ENABLE=OFF"} flag.
+
   \item In the \texttt{GNUmakefile} for the \amrex\ application which uses the Fortran 2003 interface to \cvode, add \texttt{USE\_CVODE = TRUE}, which will compile the Fortran 2003 interfaces and link the \cvode\ libraries.
   Note that one must define the \texttt{CVODE\_LIB\_DIR} environment variable to point to the location where the libraries are installed.
 \end{enumerate}
+
+
+\section{The \cvode\ Tutorials}
+
+\amrex\ provides two \cvode\ tutorials in the \texttt{Tutorials/CVODE} directory, called \texttt{EX1} and \texttt{EX2}.
+\texttt{EX1} consists of a single ODE that is integrated with \cvode\ within each cell of a 3-D grid.
+It demonstrates how to initialize the \cvode\ solver, how to call the ODE right-hand-side (RHS), and, more importantly, how to \emph{re-}initialize the solver between cells, which avoids allocating and freeing solver memory between each cell (see the call to \texttt{FCVReInit()} in the \texttt{integrate\_ode.f90} file in the \texttt{EX1} directory.)
+
+The \texttt{EX2} example demonstrates the slightly more complicated case of integrating a system of coupled ODEs within each cell.
+Similarly to \texttt{EX1}, it provides an RHS and some solver initialization.
+However, it also demonstrates the performance effect of providing an analytic Jacobian matrix for the system of ODEs, 
+rather than requiring the \cvode\ solver to compute the Jacobian matrix numerically using a finite-difference approach.
+The tutorial integrates the same system of ODEs on the same 3-D grid, but in one sweep it instructs \cvode\ 
+to use the analytic function that computes the Jacobian matrix, and in the other case, it does not,
+which requires \cvode\ to compute it manually.
+One observes a significant performance gain by providing the analytic Jacobian function.