nD nodeset: added fold axis support and --axis option to nodeset

This adds support for constrainted nD folded nodeset to improve external
tools support.

NodeSet.py:

NodeSetBase's member fold_axis can be set to an iterable over nD
0-indexed axis integers. If so, only rangesets along fold axis are
effectively folded.

However, we always use nD algorithms internally and only re-expand
rangesets of unspecified axis at __str__() time. This may lead to
suboptimal folding, mostly observed with tricky cases, like
overlapping dimensions, etc.

We cannot really talk about this as a 1.6 compat feature, as 1.6
was mostly broken on these aspects. However, for nodesets fed from
unique nD nodes, like in clush when gathering results, setting
fold_axis to [0] should lead to similar results than 1.6 (this patch
doesn't include the feature for clush).

CLI/Nodeset.py and CLI/Options.py:

Added --axis=RANGESET option to nodeset, to use with --fold. Axis are
1-indexed with nodeset CLI. Also --axis=-INDEX is supported to count
backward.

Tests, docs and examples included.

Closes #269.

Change-Id: Ib1f7947f6ecf7b4acbdd1550800a9d89e32743c4

Author: thiell

ChangeLog

@@ -1,3 +1,11 @@
+2015-08-29 S. Thiell <sthiell@stanford.edu>
+
+	* CLI/Nodeset.py: added --axis option to choose nD fold axis (ticket
+	#269).
+	* NodeSet.py: added fold_axis public member to NodeSetBase along with
+	expand algorithm when casting to string to choose nD fold axis (ticket
+	#269).
+
 2015-08-28 S. Thiell <sthiell@stanford.edu>
 
 	* CLI/Config.py: better per-user clush.conf support. clush now also checks

doc/man/man1/nodeset.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH NODESET 1 "2015-07-10" "1.7" "ClusterShell User Manual"
+.TH NODESET 1 "2015-08-29" "1.7" "ClusterShell User Manual"
 .SH NAME
 nodeset \- compute advanced nodeset operations
 .
@@ -131,6 +131,9 @@ split result into a number of subsets
 .TP
 .B \-\-contiguous
 split result into contiguous subsets (ie. for nodeset, subsets will contain nodes with same pattern name and a contiguous range of indexes, like foobar[1\-100]; for rangeset, subsets with consists in contiguous index ranges)"""
+.TP
+.BI \-\-axis\fB= RANGESET
+for nD nodesets, fold along provided axis only. Axis are indexed from 1 to n and can be specified here either using the rangeset syntax, eg. \(aq1\(aq, \(aq1\-2\(aq, \(aq1,3\(aq, or by a single negative number meaning that the indice is counted from the end. Because some nodesets may have several different dimensions, axis indices are silently truncated to fall in the allowed range.
 .UNINDENT
 .UNINDENT
 .UNINDENT
@@ -288,6 +291,14 @@ node[1\-3,5\-7]
 dc[1\-2]n[1\-2]
 .fi
 .sp
+.INDENT 7.0
+.TP
+.B $ nodeset \-\-axis=1 \-f dc1n2 dc2n2 dc1n1 dc2n1
+.UNINDENT
+.nf
+dc[1\-2]n1,dc[1\-2]n2
+.fi
+.sp
 .TP
 .B Expanding nodesets
 .INDENT 7.0

doc/sphinx/tools/nodeset.rst

@@ -483,6 +483,34 @@ contiguous node set in an expanded manner to separate lines::
     node11 node12 node13 node14 node15 node16 node17 node18 node19
 
 
+Choosing fold axis (nD)
+"""""""""""""""""""""""
+
+The default folding behavior for multidimensional node sets is to fold along
+all *nD* axis. However, other cluster tools barely support nD nodeset syntax,
+so it may be useful to fold along one (or a few) axis only. The ``--axis``
+option allows you to specify indexes of dimensions to fold. Using this
+option, rangesets of unspecified axis there won't be folded. Please note
+however that the obtained result may be suboptimal, this is because
+:class:`.NodeSet` algorithms are optimized for folding along all axis.
+``--axis`` value is a set of 0-indexed integers, representing nD axis, in the
+form of a number or a rangeset. A common case is to restrict folding on a
+single axis, like in the following simple examples::
+
+    $ nodeset --axis=0 -f node1-ib0 node2-ib0 node1-ib1 node2-ib1
+    node[1-2]-ib0,node[1-2]-ib1
+
+    $ nodeset --axis=1 -f node1-ib0 node2-ib0 node1-ib1 node2-ib1
+    node1-ib[0-1],node2-ib[0-1]
+
+Because a single nodeset may have several different dimensions, axis indices
+are silently truncated to fall in the allowed range. Negative indices are
+useful to fold along the last axis whatever number of dimensions used::
+
+    $ nodeset --axis=-1 -f comp-[1-2]-[1-36],login-[1-2]
+    comp-1-[1-36],comp-2-[1-36],login-[1-2]
+
+
 .. _nodeset-groups:
 
 Node groups

doc/txt/nodeset.txt

@@ -7,7 +7,7 @@ compute advanced nodeset operations
 -----------------------------------
 
 :Author: Stephane Thiell <sthiell@stanford.edu>
-:Date:   2015-07-10
+:Date:   2015-08-29
 :Copyright: CeCILL-C V1
 :Version: 1.7
 :Manual section: 1
@@ -72,6 +72,7 @@ OPTIONS
                         return sliced off result; examples of SLICE_RANGESET are "0" for simple index selection, or "1-9/2,16" for complex rangeset selection
     --split=MAXSPLIT    split result into a number of subsets
     --contiguous        split result into contiguous subsets (ie. for nodeset, subsets will contain nodes with same pattern name and a contiguous range of indexes, like foobar[1-100]; for rangeset, subsets with consists in contiguous index ranges)"""
+    --axis=RANGESET     for nD nodesets, fold along provided axis only. Axis are indexed from 1 to n and can be specified here either using the rangeset syntax, eg. '1', '1-2', '1,3', or by a single negative number meaning that the indice is counted from the end. Because some nodesets may have several different dimensions, axis indices are silently truncated to fall in the allowed range.
 
 
 For a short explanation of these options, see ``-h, --help``.
@@ -163,13 +164,18 @@ Folding nodesets
   | node[0-7,32-163]
 
   :$ echo node3 node6 node1 node2 node7 node5 | nodeset -f:
-
+  
   | node[1-3,5-7]
 
   :$ nodeset -f dc1n2 dc2n2 dc1n1 dc2n1:
 
   | dc[1-2]n[1-2]
 
+  :$ nodeset --axis=1 -f dc1n2 dc2n2 dc1n1 dc2n1:
+  
+  | dc[1-2]n1,dc[1-2]n2
+
+
 
 Expanding nodesets
   :$ nodeset -e node[160-163]:

lib/ClusterShell/CLI/Nodeset.py

@@ -195,6 +195,9 @@ def nodeset():
     if options.maxsplit is None:
         options.maxsplit = 1
 
+    if options.axis and (not options.fold or options.rangeset):
+        parser.error("--axis option is only supported when folding nodeset")
+
     if options.groupsource and not options.quiet and class_set == RangeSet:
         print >> sys.stderr, "WARNING: option group source \"%s\" ignored" \
                                 % options.groupsource
@@ -277,6 +280,15 @@ def nodeset():
         autofactor = float(options.autostep)
         xset.autostep = int(math.ceil(float(len(xset)) * autofactor))
 
+    # user-specified nD-nodeset fold axis
+    if options.axis:
+        if not options.axis.startswith('-'):
+            # axis are 1-indexed in nodeset CLI (0 ignored)
+            xset.fold_axis = tuple(x - 1 for x in RangeSet(options.axis) if x > 0)
+        else:
+            # negative axis index (only single number supported)
+            xset.fold_axis = [int(options.axis)]
+
     fmt = options.output_format # default to '%s'
 
     # Display result according to command choice
@@ -304,12 +316,9 @@ def main():
     """main script function"""
     try:
         nodeset()
-    except AssertionError, ex:
+    except (AssertionError, IndexError, ValueError), ex:
         print >> sys.stderr, "ERROR:", ex
         sys.exit(1)
-    except IndexError:
-        print >> sys.stderr, "ERROR: syntax error"
-        sys.exit(1)
     except SyntaxError:
         print >> sys.stderr, "ERROR: invalid separator"
         sys.exit(1)

lib/ClusterShell/CLI/OptionParser.py

@@ -321,4 +321,7 @@ def install_nodeset_options(self):
         optgrp.add_option("--contiguous", action="store_true",
                           dest="contiguous", help="split result into "
                                                   "contiguous subsets")
+        optgrp.add_option("--axis", action="store", dest="axis",
+                          metavar="RANGESET", help="fold along these axis only "
+                                                   "(axis 1..n for nD nodeset)")
         self.add_option_group(optgrp)

lib/ClusterShell/NodeSet.py

@@ -134,11 +134,12 @@ class NodeSetBase(object):
        'node[1-5,7]-ib[1-2]'
     """
     def __init__(self, pattern=None, rangeset=None, copy_rangeset=True,
-                 autostep=None):
+                 autostep=None, fold_axis=None):
         """New NodeSetBase object initializer"""
         self._autostep = autostep
         self._length = 0
         self._patterns = {}
+        self.fold_axis = fold_axis
         if pattern:
             self._add(pattern, rangeset, copy_rangeset)
         elif rangeset:
@@ -252,6 +253,49 @@ def __len__(self):
                 cnt += 1
         return cnt
 
+    def _iter_nd_pat(self, pat, rset):
+        """
+        Take a pattern and a RangeSetND object and iterate over nD computed
+        nodeset strings while following fold_axis constraints.
+        """
+        try:
+            dimcnt = rset.dim()
+            if self.fold_axis is None:
+                # fold along all axis (default)
+                fold_axis = range(dimcnt)
+            else:
+                # set of user-provided fold axis (support negative numbers)
+                fold_axis = [int(x) % dimcnt for x in self.fold_axis
+                             if -dimcnt <= int(x) < dimcnt]
+        except (TypeError, ValueError), exc:
+            raise NodeSetParseError("fold_axis=%s" % self.fold_axis, exc)
+
+        for rgvec in rset.vectors():
+            rgnargs = []    # list of str rangeset args
+            for axis, rangeset in enumerate(rgvec):
+                # build an iterator over rangeset strings to add
+                if len(rangeset) > 1:
+                    if axis not in fold_axis: # expand
+                        rgstrit = rangeset.striter()
+                    else:
+                        rgstrit = ["[%s]" % rangeset]
+                else:
+                    rgstrit = [str(rangeset)]
+
+                # aggregate/expand along previous computed axis...
+                t_rgnargs = []
+                for rgstr in rgstrit: # 1-time when not expanding
+                    if not rgnargs:
+                        t_rgnargs.append([rgstr])
+                    else:
+                        for rga in rgnargs:
+                            t_rgnargs.append(rga + [rgstr])
+                rgnargs = t_rgnargs
+
+            # get nodeset patterns formatted with range strings
+            for rgargs in rgnargs:
+                yield pat % tuple(rgargs)
+
     def __str__(self):
         """Get ranges-based pattern of node list."""
         results = []
@@ -260,29 +304,27 @@ def __str__(self):
                 if not rset:
                     results.append(pat % ())
                 elif rset.dim() == 1:
-                    rgs = str(rset)
-                    cnt = len(rset)
-                    if cnt > 1:
-                        rgs = "[%s]" % rgs
-                    results.append(pat % rgs)
+                    # check if allowed to fold even for 1D pattern
+                    if self.fold_axis is None or \
+                            list(x for x in self.fold_axis if -1 <= int(x) < 1):
+                        rgs = str(rset)
+                        cnt = len(rset)
+                        if cnt > 1:
+                            rgs = "[%s]" % rgs
+                        results.append(pat % rgs)
+                    else:
+                        results.extend((pat % rgs for rgs in rset.striter()))
                 elif rset.dim() > 1:
-                    for rgvec in rset.vectors():
-                        rgargs = []
-                        for rangeset in rgvec:
-                            rgs = str(rangeset)
-                            cnt = len(rangeset)
-                            if cnt > 1:
-                                rgs = "[%s]" % rgs
-                            rgargs.append(rgs)
-                        results.append(pat % tuple(rgargs))
+                    results.extend(self._iter_nd_pat(pat, rset))
         except TypeError:
-            raise NodeSetParseError(pat, "Internal error: " \
-                                         "node pattern and ranges mismatch")
+            raise NodeSetParseError(pat, "Internal error: node pattern and "
+                                         "ranges mismatch")
         return ",".join(results)
 
     def copy(self):
         """Return a shallow copy."""
         cpy = self.__class__()
+        cpy.fold_axis = self.fold_axis
         cpy._autostep = self._autostep
         cpy._length = self._length
         dic = {}
@@ -1071,33 +1113,45 @@ class NodeSet(NodeSetBase):
 
     _VERSION = 2
 
-    def __init__(self, nodes=None, autostep=None, resolver=None):
+    def __init__(self, nodes=None, autostep=None, resolver=None,
+                 fold_axis=None):
         """Initialize a NodeSet object.
 
-        The `nodes' argument may be a valid nodeset string or a NodeSet
+        The `nodes` argument may be a valid nodeset string or a NodeSet
         object. If no nodes are specified, an empty NodeSet is created.
 
-        The optional `autostep' argument is passed to underlying RangeSet
-        objects and aims to enable and make use of the range/step syntax
-        (eg. node[1-9/2]) when converting NodeSet to string (using folding).
-        To enable this feature, autostep must be set there to the min number of
-        indexes that are found at equal distance of each other inside a range
-        before NodeSet starts to use this syntax. For example, autostep=3 (or
-        less) will pack n[2,4,6] into n[2-6/2]. Default autostep value is None
-        which means "inherit whenever possible", ie. do not enable it unless
-        set in NodeSet objects passed as `nodes' here or during arithmetic
-        operations.
-        You may however use the special AUTOSTEP_DISABLED constant to force
+        The optional `autostep` argument is passed to underlying
+        :class:`.RangeSet.RangeSet` objects and aims to enable and make use of
+        the range/step syntax (eg. ``node[1-9/2]``) when converting NodeSet to
+        string (using folding). To enable this feature, autostep must be set
+        there to the min number of indexes that are found at equal distance of
+        each other inside a range before NodeSet starts to use this syntax. For
+        example, `autostep=3` (or less) will pack ``n[2,4,6]`` into
+        ``n[2-6/2]``. Default autostep value is None which means "inherit
+        whenever possible", ie. do not enable it unless set in NodeSet objects
+        passed as `nodes` here or during arithmetic operations.
+        You may however use the special ``AUTOSTEP_DISABLED`` constant to force
         turning off autostep feature.
 
-        The optional `resolver' argument may be used to override the group
+        The optional `resolver` argument may be used to override the group
         resolving behavior for this NodeSet object. It can either be set to a
-        GroupResolver object, to the RESOLVER_NOGROUP constant to disable any
-        group resolution, or to None (default) to use standard NodeSet group
-        resolver (see set_std_group_resolver() at the module level to change
-        it if needed).
-        """
-        NodeSetBase.__init__(self, autostep=autostep)
+        :class:`.NodeUtils.GroupResolver` object, to the ``RESOLVER_NOGROUP``
+        constant to disable any group resolution, or to None (default) to use
+        standard NodeSet group resolver (see :func:`.set_std_group_resolver()`
+        at the module level to change it if needed).
+
+        nD nodeset only: the optional `fold_axis` parameter, if specified, set
+        the public instance member `fold_axis` to an iterable over nD 0-indexed
+        axis integers. This parameter may be used to disengage some nD folding.
+        That may be useful as all cluster tools don't support folded-nD nodeset
+        syntax. Pass ``[0]``, for example, to only fold along first axis (that
+        is, to fold first dimension using ``[a-b]`` rangeset syntax whenever
+        possible). Using `fold_axis` ensures that rangeset won't be folded on
+        unspecified axis, but please note however, that using `fold_axis` may
+        lead to suboptimial folding, this is because NodeSet algorithms are
+        optimized for folding along all axis (default behavior).
+        """
+        NodeSetBase.__init__(self, autostep=autostep, fold_axis=fold_axis)
 
         # Set group resolver.
         if resolver in (RESOLVER_NOGROUP, RESOLVER_NOINIT):
@@ -1155,6 +1209,7 @@ def __setstate__(self, dic):
         self._resolver = None
         self._parser = ParsingEngine(None)
         if getattr(self, '_version', 1) <= 1:
+            self.fold_axis = None
             # if setting state from first version, a conversion is needed to
             # support native RangeSetND
             old_patterns = self._patterns
@@ -1179,6 +1234,7 @@ def copy(self):
             else:
                 dic[pat] = rangeset.copy()
         cpy._patterns = dic
+        cpy.fold_axis = self.fold_axis
         cpy._autostep = self._autostep
         cpy._resolver = self._resolver
         cpy._parser = self._parser