Update the ULFM Readme

Signed-off-by: Aurelien Bouteiller <bouteill@icl.utk.edu>

Author: abouteiller

docs/features/ulfm.rst

@@ -6,6 +6,17 @@ User-Level Fault Mitigation (ULFM)
 This chapter documents the features and options specific to the **User
 Level Failure Mitigation (ULFM)** Open MPI implementation.
 
+TL;DR
+-----
+This is an extremely terse summary of how to use ULFM:
+
+.. code-block::
+
+  ./configure --with-ft=ulfm [...options...]
+  make [-j N] all install
+  mpicc my-ft-program.c -o my-ft-program
+  mpiexec -n 4 --with-ft ulfm my-ft-program
+
 Features
 --------
 
@@ -100,11 +111,11 @@ Available from: https://journals.sagepub.com/doi/10.1177/1094342013488238.
 Building ULFM support in Open MPI
 ---------------------------------
 
-In Open MPI |ompi_ver|, ULFM support is **enabled by default** |mdash|
-when you build Open MPI, unless you specify ``--without-ft``, ULFM
+In Open MPI |ompi_ver|, ULFM support is **built-in by default** |mdash|
+that is, when you build Open MPI, unless you specify ``--without-ft``, ULFM
 support will automatically be built.
 
-Optionally, you can specify ``--with-ft`` to ensure that ULFM support
+Optionally, you can specify ``--with-ft ulfm`` to ensure that ULFM support
 is definitely built.
 
 Support notes
@@ -215,7 +226,7 @@ Running your application
 
 You can launch your application with fault tolerance by simply using
 the normal Open MPI ``mpiexec`` launcher, with the
-``--with-ft ulfm`` CLI option:
+``--with-ft ulfm`` CLI option (or its synonym ``--with-ft mpi``):
 
 .. code-block::
 
@@ -234,6 +245,11 @@ you use ``mpiexec`` within an allocation (e.g., ``salloc``,
 Run-time tuning knobs
 ^^^^^^^^^^^^^^^^^^^^^
 
+The main control for enabling/disabling fault tolerance at runtime
+is the ``--with-ft ulfm`` (or its synomym ``--with-ft mpi``) ``mpiexec``
+CLI option. This option will setup multiple subsystems of Open MPI
+to enable fault tolerance.
+
 ULFM comes with a variety of knobs for controlling how it runs. The
 default parameters are sane and should result in good performance in
 most cases. You can change the default settings with ``--mca
@@ -243,9 +259,10 @@ errmgr_detector_bar <value>`` for PRTE options.
 PRTE level options
 ~~~~~~~~~~~~~~~~~~
 
-* ``prrte_enable_recovery <true|false> (default: false)`` controls
+* ``prrte_enable_ft <true|false> (default: false)`` controls
   automatic cleanup of apps with failed processes within
-  mpirun. Enabling this option also enables ``mpi_ft_enable``.
+  mpirun. This option is automatically set to ``true`` when using
+  ``--with-ft ulfm``.
 * ``errmgr_detector_priority <int> (default 1005``) selects the
   PRRTE-based failure detector. Only available when
   ``prte_enable_recovery`` is ``true``. You can set this to ``0`` when
@@ -263,17 +280,29 @@ PRTE level options
 Open MPI level options
 ~~~~~~~~~~~~~~~~~~~~~~
 
-* ``mpi_ft_enable <true|false> (default: same as
-  prrte_enable_recovery)`` permits turning on/off fault tolerance at
-  runtime. When false, failure detection is disabled; Interfaces
-  defined by the fault tolerance extensions are substituted with dummy
-  non-fault tolerant implementations (e.g., ``MPIX_Comm_agree`` is
-  implemented with ``MPI_Allreduce``); All other controls below become
-  irrelevant.
+Some default values are applied to some Open MPI parameters when using
+``mpiexec --with-ft ulfm``. These defaults are obtained from the ``ft-mpi``
+aggregate MCA param file
+``$installdir/share/openmpi/amca-param-sets/ft-mpi``. You can tune the
+runtime behavior with ULFM by either setting or unsetting variables in
+this file, or by overiding the variable on the command line (e.g.,
+``--mca btl ofi,self``). Note that if fault tolerance is not enabled at
+runtime (that is, when not using ``--with-ft ulfm``), this param file is
+not loaded, which may change which components are selected (this in turn
+may change observed performance when comparing with and without fault
+tolerance).
+
+* ``mpi_ft_enable <true|false> (default: false)``
+  permits turning on/off fault tolerance at runtime. This option is
+  automatically set to ``true`` from the aggregate MCA param file
+  ``ft-mpi`` loaded when using ``--with-ft ulfm``. When false, failure
+  detection is disabled; Interfaces defined by the fault tolerance extensions
+  are substituted with dummy non-fault tolerant implementations (e.g.,
+  ``MPIX_Comm_agree`` is implemented with ``MPI_Allreduce``); All other
+  controls below become irrelevant.
 * ``mpi_ft_verbose <int> (default: 0)`` increases the output of the
   fault tolerance activities. A value of 1 will report detected
-  failures.
-* ``mpi_ft_detector <true|false> (default: false)``, **EXPERIMENTAL**
+  failuresulfm ``mpi_ft_detector <true|false> (default: false)``, **DEPRECATED**
   controls the activation of the Open MPI level failure detector. When
   this detector is turned off, all failure detection is delegated to
   PRTE (see above).  The Open MPI level fault detector is
@@ -291,13 +320,16 @@ Open MPI level options
   latency (typically 1us increase).* You may want to **enable this
   option if you experience false positive** processes incorrectly
   reported as failed with the Open MPI failure detector.
+  This option is only relevant when `mpi_ft_detector` is `true`.
 * ``mpi_ft_detector_period <float> (default: 3e0 seconds)`` heartbeat
   period. Recommended value is 1/3 of the timeout. _Values lower than
   100us may impart a noticeable effect on latency (typically a 3us
   increase)._
+  This option is only relevant when `mpi_ft_detector` is `true`.
 * ``mpi_ft_detector_timeout <float> (default: 1e1 seconds)`` heartbeat
   timeout (i.e. failure detection speed). Recommended value is 3 times
   the heartbeat period.
+  This option is only relevant when `mpi_ft_detector` is `true`.
 
 Known Limitations in ULFM
 ^^^^^^^^^^^^^^^^^^^^^^^^^