Adds documentation

doc/source/users_guide/machine.rst

@@ -57,9 +57,35 @@ Each ``<machine>`` tag requires the following input:
   * May have optional attributes of ``compiler``, ``mpilib`` and/or ``threaded``
   * May have an optional ``<arguments>`` element which in turn contains one or more ``<arg>`` elements.
     These specify the arguments to the mpi executable and are dependent on your mpi library implementation.
-  * May have an option ``<run_exe>`` element which overrides the ``default_run_exe``
-  * May have an option ``<run_misc_suffix>`` element which overrides the ``default_run_misc_suffix``
+  * May have an optional ``<run_exe>`` element which overrides the ``default_run_exe``
+  * May have an optional ``<run_misc_suffix>`` element which overrides the ``default_run_misc_suffix``
+  * May have an optional ``<aprun_mode>`` element which controls how CIME generates arguments when ``<executable>`` contains ``aprun``. 
 
+  The ``<aprun_mode>`` element can be one of the following. The default value is ``ignore``.
+
+  * ``ignore`` will cause CIME to ignore it's aprun module and join the values found in ``<arguments>``.
+  * ``default`` will use CIME's aprun module to generate arguments.
+  * ``override`` behaves the same as ``default`` expect it will use ``<arguments>`` to mutate the generated arguments. When using this mode a ``position`` attribute can be placed on ``<arg>`` tags to specify how it's used.
+
+  The ``position`` attribute on ``<arg>`` can take one of the following values. The default value is ``per``.
+
+  * ``global`` causes the value of the ``<arg>`` element to be used as a global argument for ``aprun``.
+  * ``per`` causes the value of the ``<arg>`` element to be appended to each separate binaries arguments.
+
+  Example using ``override``:
+  ::
+
+    <executable>aprun</executable>
+    <aprun_mode>override</aprun_mode>
+    <arguments>
+      <arg position="global">-e DEBUG=true</arg>
+      <arg>-j 20</arg>
+    </arguments>
+
+  Sample command output:
+  ::
+
+    aprun -e DEBUG=true ... -j 20 e3sm.exe : ... -j 20 e3sm.exe
 
 * ``module_system``: How and what modules to load on this system. Module systems allow you to easily load multiple compiler environments on a machine. CIME provides support for two types of module tools: `module <http://www.tacc.utexas.edu/tacc-projects/mclay/lmod>`_ and `soft  <http://www.mcs.anl.gov/hs/software/systems/softenv/softenv-intro.html>`_. If neither of these is available on your machine, simply set ``<module_system type="none"\>``.