mcxassemble.azm

\import{mcx.zmm}

\begin{pud::man}{

   {name}{mcxassemble}
   {html_title}{The mcxassemble manual}
   {author}{Stijn van Dongen}
   {section}{1}
   {synstyle}{long}
   {defstyle}{long}

   \man_share
}

\${html}{\"pud::man::maketoc"}

\sec{name}{NAME}
\NAME{mcxassemble}{transform raw cooccurrence data to mcl matrix format.}

\sec{synopsis}{SYNOPSIS}
\par{
\mcxassemble
\synreqopt{-b}{base}{base name}
\synoptopt{-o}{fname}{write to file fname}
\synoptopt{--write-binary}{write output in binary format}
\synoptopt{--map}{apply base.map}
\synoptopt{-raw-tf}{apply transform spec to input}
\synoptopt{-rv}{MODE}{repeated vectors}
\synoptopt{-re}{MODE}{repeated entries}
\synoptopt{-ri}{MODE}{adding mirror image}
\synoptopt{-r}{MODE}{repeated entries/vectors/images}
\synoptopt{-prm-tf}{apply transform spec to primary matrix}
\synoptopt{-sym-tf}{apply transform spec to symmetrified matrix}
\synoptopt{-q}{quiet mode}
   }

\par{
   The options above embody the default setup when using mcxassemble.
   There are many more options which mostly provide subtly different
   ways of doing input/output, set warning levels, or regulate
   how repeated entries and vectors should be treated.
   The full list of options is shown below.
   Read \secref{description} for learning about mcxassemble input/output
   and the functionality it provides.
   }

\cpar{NOTE}{
   As of release 05-314 \mysib{mcl} is able to cluster label-type input
   on the fly. In most cases, this will be sufficient.  Alternatively,
   \mysib{mcxload} can be used to map label-type input onto mcl
   matrices. Consequently, there are likely fewer scenarios nowadays
   where \mcxassemble is the best solution. Consider first whether
   \mcl in label mode or \mcxload can do the job as well.}

\par{
\mcxassemble
\synoptopt{-b}{base}{base name}
\synoptopt{-hdr}{fname}{read header file}
\synoptopt{-raw}{fname}{read raw file}
\synoptopt{--map}{apply base.map}
\synoptopt{--cmap}{apply base.cmap}
\synoptopt{--rmap}{apply base.rmap}
\synoptopt{-map}{fname}{apply fname}
\synoptopt{-rmap}{fname}{apply fname}
\synoptopt{-cmap}{fname}{apply fname}
\synoptopt{-tag}{tag}{apply base.tag}
\synoptopt{-rtag}{tag}{apply base.tag}
\synoptopt{-ctag}{tag}{apply base.tag}
\synoptopt{-skw}{fname}{write skew matrix}
\synoptopt{-prm}{fname}{write primary result matrix}
\synoptopt{--skw}{write base.skw}
\synoptopt{--prm}{write base.prm}
\synoptopt{-xo}{suf}{write base.suf}
\synoptopt{-o}{fname}{write to file fname}
\synoptopt{-n}{do not write default symmetrized result}
\synoptopt{-i}{read from single data file}
\synoptopt{-digits}{int}{digits width}
\synoptopt{-s}{check for symmetry}
\synoptopt{-raw-tf}{apply transform spec to input}
\synoptopt{-rv}{<mode>}{action for repeated vectors}
\synoptopt{-re}{<mode>}{action for repeated entries}
\synoptopt{-ri}{<mode>}{adding mirror image}
\synoptopt{-r}{<mode>}{same for entries and vectors}
\synoptopt{-prm-tf}{apply transform spec to primary matrix}
\synoptopt{-sym-tf}{apply transform spec to symmetrified matrix}
\synoptopt{--quiet-re}{quiet for repeated entries}
\synoptopt{--quiet-rv}{quiet for repeated vectors}
\synoptopt{-q}{the two above combined}
\stdsynopt
   }

\sec{description}{DESCRIPTION}

\par{
   \mcxassemble enables easy matrix creation from an intermediate raw matrix
   format that can easily be constructed from a one-pass-parse of cooccurrence
   data. The basic setup is as follows.
   }

\begin{itemize}{
   {mark}{\*{itembullet}}
   {flow}{compact}
   {interitem}{0}
   {align}{right}
}
\item
\car{
   Parse cooccurrence data from some external format.
   }

\item
\car{
   Transform cooccurrence data to raw mcl data as you parse.
   }

\item
\car{
   When done, write out required header and domain information
   to a separate file. The domain information can be built during
   the parsing stage.
   }

\item
\car{
   Use mcxassemble to construct a valid matrix from the raw data
   and the header information.
   }
\item
\car{
   Nodes can optionally be relabeled by writing a separate map file to be read
   by \mcxassemble, which takes the form of a very thin matrix file.
   }
\end{itemize}

\par{
   The easiest thing to do is to group all input/output files under the same
   base name, say\~\bf{base}. A standard way of proceeding, which will lead to
   a concise \mcxassemble command line, is by creating the input files
   \bf{base.raw} and \bf{base.hdr}, and optionally the file \bf{base.map}.  The
   default behaviour of mcxassemble is then to create \bf{base.sym} as the
   resulting matrix file, containing the symmetrized matrix constructed from
   the raw input.
   }

\cpar{Example}{
   Suppose \v{blastresult} is a file containing blast results.
   The following two commands construct an mcl matrix file from the file.
   }

\verbatim{\
   mcxdeblast --score=e --sort=a blastresult
   mcxassemble -b blastresult -r max --map}

\car{
   \mcxdeblast will generate the
   files \v{blastresult.hdr}, \v{blastresult.raw}, and \v{blastresult.map}.
   The \useopt{--sort=a} option will create a map file corresponding
   with alphabetic ordering. These files are processed by \mcxassemble
   and it will generate the file \v{blastresult.sym}. The \useopt{-r}
   option tells \mcxassemble that repeated entries should be maxed;
   each time the largest entry seen thus far will be taken.
   }

\cpar{Header file}{
   This file contains a header as usually found in generic mcl matrix files,
   i.e. the required \it{header} part, and optionally the \it{domain} part(s)
   if not all domains are canonical.  Refer to \mysib{mcxio} for more information.
   The domain information in the header file will be used to pre-construct a
   skeleton matrix and to validate the entries in the raw data file as they
   fill the skeleton matrix.
   }

\cpar{Raw input format}{
   The file from which raw input is read should have the raw format as
   described in \mysib{mcxio}. Simply put; no header specification, no domain
   specification, and no matrix introduction syntax is used. The file just
   contains a listing of vectors.  An example fragment is the following:
   }

\verbatix{\
2  4:0.34 1:2.8838 4:2.328 1:4.238 1:12 $
1  2:7.8 $
2  1:0.01 4:20.3 3:2 $}

\car{
   The listing of vectors need not be sorted, and neither does
   a vector itself need to be sorted - the mcl generic matrix format
   is actually not different in this respect.
   Furthermore, duplicate entries and duplicate vectors are allowed.
   This is in fact again allowed in the generic format, except
   that where applications expect generic format warnings will be issued and
   duplicate entries will be disregarded.  \mcxassemble allows customizable
   behaviour dictating how to merge repeated entries.
   Refer to the \optref{-re}{\genopt{-re},\~\genopt{-rv},\~\genopt{-r}}
   options below.
   }

\par{
   The vectors read by \mcxassemble do have to match the domains specified in
   the header file. The leading index that specifies the column index has to be
   present in the column domain; all subsequent indices that specify column
   entries have to be present in the row domain.
   }

\par{
   \it{If one concatenates the contents of the header file and the data file},
   the result is \it{almost but not quite} a file containing a matrix in
   syntactically correct mcl generic matrix format. The parts missing
   are the \v{(mclmatrix} introduction token, (followed by) the
   \v{begin} token, and the closing \v{)} token.
   }

\cpar{Map file}{
   This file must contain a map matrix, which is a matrix with the
   following properties:
   }

\begin{itemize}{
   {mark}{\*{itembullet}}
   {flow}{compact}
   {interitem}{0}
   {align}{right}
}
\item
\car{
   The column domain and row domain are of the same cardinality.
   }
\item
\car{
   Each column has exactly one entry.
   }
\item
\car{
   Each row domain index occurs in exactly one column.
   }
\end{itemize}

\par{
   Such a matrix is used to relabel the nodes as found in the raw data.  A
   situation that might occur when parsing some external format (and producing
   raw matrix format), is that ID's (indices) are handed out on the fly during
   the parse.  Afterwards, one may want to relabel the IDs such that they
   correspond with an alphabetic listing of the quantity that is represented by
   the node domain, or by some other sort criterion.  A map file is then
   typically generated by the parser, as that is the utility in charge of the
   IDs.  A small example of a map file for a graph containing five nodes is the
   following:
   }

\verbatim{\
(mclheader
mcltype matrix
dimensions 5x5
)
(mclmatrix
begin
0  4  $  #  mno 
1  2  $  #  ghi
2  1  $  #  def
3  3  $  #  jkl
4  0  $  #  abc
)}

\car{
   This corresponds to a relabeling such that the associated strings
   will be ordered alphabetically. Note that comments can be used
   to link string identifiers with indices. This map file says e.g. that
   the string identifier "mno" is represented by index 0 in the raw data,
   and by index 4 in the matrix output by \mcxassemble.
   }

\sec{options}{OPTIONS}
\begin{itemize}{
   {flow}{cascade}
   {interitem}{1}
   \mcx_itemopts
}

\items{
   {\defopt{-b}{base}{base name}}
}
\car{
   Base name of files to be processed and output. Refer to \secref{description}
   above and the entries of other options below.
   }

\items{
   {\defopt{-hdr}{fname}{read header file}}
   {\defopt{-raw}{fname}{read raw file}}
}
\car{
   Explicitly specify the header file and the data file (rather
   than constructing the file names from a base name and suffixes).
   }

\items{
   {\defopt{--map}{apply base.map}}
   {\defopt{--cmap}{apply base.cmap}}
   {\defopt{--rmap}{apply base.rmap}}
   {\defopt{-map}{fname}{apply fname}}
   {\defopt{-rmap}{fname}{apply fname}}
   {\defopt{-cmap}{fname}{apply fname}}
   {\defopt{-tag}{tag}{apply base.tag}}
   {\defopt{-rtag}{tag}{apply base.tag}}
   {\defopt{-ctag}{tag}{apply base.tag}}
}
\car{
   Map options. \useopt{--cmap} combines with the \useopt{-b}\~option,
   and says that the map file in \bf{base}.\v{cmap} (where \bf{base}
   was specified with \useopt{-b}{base}) should be applied to the column
   domain only. \useopt{--rmap} works the same for the
   row domain, and \useopt{--map} can be used to apply the same map
   to both the column and row domains.
   }

\par{
   \useopt{-cmap} and its siblings are used to explicitly specify the 
   map file to be used, rather than combining a base name with a fixed
   suffix.
   \useopt{-tag} and its siblings work in conjuction with
   the \useopt{-b}\~option, and require that a tag be specified from
   which to construct the map file (by appending it to the base name).
   }

\items{
   {\defopt{-skw}{fname}{write skew matrix}}
   {\defopt{-prm}{fname}{write primary result matrix}}
   {\defopt{--prm}{write base.prm}}
   {\defopt{--skw}{write base.skw}}
   {\defopt{-n}{do not write default symmetrized result}}
}
\car{
   Options for writing matrices other than the default symmetrized result.
   The primary result matrix is the matrix constructed from reading in the
   raw data and adding entries to the skeleton matrix as specified
   with the \optref{-r}{\useopt{-r}, \useopt{-re}, and \useopt{-rv} options}.
   This matrix can be written using one of the \bf{prm} options.
   Calling the primary matrix A, the skew matrix (as defined here)
   is the matrix \v{A\~-\~A^T}, i.e. A minus its transposed matrix.
   It can be written using one of the \bf{skw} options.
   }

\par{
   If for some reason the symmetrized result is not needed, its output
   can be prevented using the \genopt{-n}\~option.
   }

\items{
   {\defopt{-xo}{suf}{write base.suf}}
   {\defopt{-o}{fname}{write to file fname}}
   {\defopt{-i}{read from single data file}}
   {\defopt{-digits}{int}{digits width}}
   {\defopt{--write-binary}{write output in binary format}}
}
\car{
   The \useopt{-xo}\~option is used in conjunction with the \useopt{-b}\~option
   in order to change the suffix for the file in which the symmetrized
   result matrix is written. Use e.g. \useopt{-xo}{mci} to change the suffix
   from the default value \v{sym} to \v{mci}.  Use \useopt{-o} to explicitly
   specify the filename in full.  Use \useopt{-digits} to set the number of
   digits written for matrix entries (c.q. edge weights).
   }

\par{
   The \useopt{-i} option is special. It causes
   \mcxassemble to read both the header information and the raw data
   from the same file, where the syntax should be fully conforming
   to generic mcl matrix format.
   }

\item{\defopt{-s}{check for symmetry}}
\car{
   This will check whether the primary result matrix was symmetric.
   It reports the number of failing (or \it{skew}) edges.
   }

\items{
   {\defopt{-raw-tf}{<tf-spec>}{apply transform spec to input}}
   {\defopt{-prm-tf}{apply transform spec to primary matrix}}
   {\defopt{-sym-tf}{apply transform spec to symmetrified matrix}}
}
\car{
   The first applies its transformation spec to the values
   as found in the raw data. The second applies its transformation
   spec to the primary matrix. The third applies its transformation
   step to the symmetrified matrix.
   Refer to \mysib{mcxio} for documentation on the transformation
   spec syntax.} 

\items{
   {\defopt{-rv}{add|max|min|mul|left|right}{action for repeated vectors}}
   {\defopt{-re}{add|max|min|mul|left|right}{action for repeated entries}}
   {\defopt{-ri}{add|max|min|mul}{adding mirror image}}
   {\defopt{-r}{add|max|min|mul|left|right}{same for entries and vectors}}
}
\car{
   Merge options, dictating the behaviour when repeated entries are
   found. A distinction is made between entries that are repeated within
   the same column listing, and entries that are repeated between
   different column listings. An entry can be a repeat of both kinds
   simultaneously as well.
   Additionally, the final result is by default symmetrized by combining with
   the mirror image (in matrix terminology, the \it{transposed} matrix).  This
   symmetrization can be done in the same variety of ways.
   }

\par{
   The \bf{re} option, for repeats within the same column, is carried out
   first. It is applied \it{after} the column has its entries sorted, so the
   \v{left} and \v{right} options are not garantueed to follow the order found
   in the raw input.  The \bf{rv} option, for repeats over different columns,
   is carried out second.
   }

\par{
   The option \useopt{-ri}{min} can assist in implementing
   a (top-list) best reciprocal hit criterion.

   }

\cpar{Examples}{
   The column
   }

\verbatim{0 1:30 1:50 2:60 4:70 3:20 1:40 2:40 $}

\car{
   is encountered in the input, listing entries for the vector labeled
   with index\~\m{0}. If \useopt{-re}{add} or \useopt{-r}{add}
   is used, it will transform to the vector
   }

\verbatim{0 1:120 2:60  3:20 4:70 $}

\car{
   If \useopt{-re}{max} or \useopt{-r}{add}
   is used instead, it will transform to the vector
   }

\verbatim{0 1:40 2:60 3:20 4:70 $}

\car{
   Suppose \genarg{add} mode is used, and that later on another
   vector specification for the index\~\m{0} is found, leading
   to this transformed vector:
   }

\verbatim{0 1:60 2:80 4:40 $}

\car{
   If \useopt{-rv}{max} was specified, this new vector is combined with the
   previous vector by taking the entry wise maximum:
   }

\verbatim{\
0 1:120 2:60 3:20 4:70 $      # first (transformed) vector
0 1:60 2:80 4:40 $            # second vector

0 1:120 2:80 3:20 4:70 $      # entry wise maximum}

\car{
   Finally, suppose that somewhere one or more vector listings
   were specified for index\~\m{3}, which eventually led to an entry \m{0:50}.
   The final symmetrization step will take the \m{[0,3]}
   entry of weight\~\m{20} and combine it with the \m{[3,0]} entry
   of weight\~\m{50}. The resulting matrix will then have the \m{[0,3]}
   and the \m{[3,0]} entry both equal to either the maximum, the sum,
   or the product of the two quantities\~\m{50} and\~\m{20}.
   }


\items{
   {\defopt{--quiet-re}{quiet for repeated entries}}
   {\defopt{--quiet-rv}{quiet for repeated vectors}}
   {\defopt{-q}{the two above combined}}
}
\car{
   Warning options. Turn these on if you expect the raw data to be free
   of repeats.
   }

\stddefopt

\end{itemize}


\sec{author}{AUTHOR}
Stijn van Dongen.

\sec{seealso}{SEE ALSO}
\car{
   \mysib{mcxio}, \mysib{mcl}, \mysib{mcxload}
   and \mysib{mclfamily} for an overview of all the documentation
   and the utilities in the mcl family.
   }

\end{pud::man}