Merge branch 'master' into v8-stable

Conflicts:
	source/conf.py

Author: rgerhards

source/conf.py

@@ -44,16 +44,16 @@
 
 # General information about the project.
 project = u'rsyslog'
-copyright = u'2008-2014, Rainer Gerhards and Adiscon'
+copyright = u'2008-2015, Rainer Gerhards and Adiscon'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
 # The short X.Y version.
-version = '8.7'
+version = '8.8'
 # The full version, including alpha/beta/rc tags.
-release = '8.7.0'
+release = '8.8.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

source/configuration/modules/imfile.rst

@@ -36,15 +36,27 @@ last processed location and continues to work from there upon restart.
 So no data is lost during a restart (except, as noted above, if the file
 is rotated just in this very moment).
 
-Currently, the file must have a fixed name and location (directory).
-It is planned to add support for dynamically generating file names in the
-future.
-
 See Also
 ........
 
 * presentation on `using wildcards with imfile <http://www.slideshare.net/rainergerhards1/using-wildcards-with-rsyslogs-file-monitor-imfile>`_
 
+Meta Data
+.........
+The imfile module supports message metadata. It supports the following
+data items
+
+- filename 
+
+  Name of the file where the message originated from. This is most
+  useful when using wildcards inside file monitors, because it then
+  is the only way to know which file the message originated from.
+
+Meta data is only present if enabled. By default it is enabled for
+input() statements that contain wildcards. For all others, it is
+disabled by default. It can explicitely be turned on or off via the
+*addMetadata* input() parameter, which always overrides the default.
+
 State Files
 ...........
 Rsyslog must keep track of which parts of the to be monitored file
@@ -132,7 +144,8 @@ Input Parameters
 
    **(Required Parameter)**
    The file being monitored. So far, this must be an absolute name (no
-   macros or templates)
+   macros or templates). Note that wildcards are supported at the file
+   name level (see "Wildcards" above for more details).
 
 .. index:: 
    single: imfile; Tag
@@ -266,6 +279,13 @@ Input Parameters
 
    Binds the listener to a specific :doc:`ruleset <../../concepts/multi_ruleset>`.
 
+.. function:: addMetadata [on/off]
+
+   **Default: see intro section on Metadata**
+
+   This is used to turn on or off the addition of metadata to the
+   message object.
+
 .. function:: stateFile [name-of-state-file]
 
    **Default: unset**

source/configuration/modules/imjournal.rst

@@ -1,7 +1,7 @@
 imjournal: Systemd Journal Input Module
 =======================================
 
-**Module Name:    imjournal**
+**Module Name:** imjournal
 
 **Author:** Milan Bartos <mbartos@redhat.com> (This module is **not**
 project-supported)
@@ -80,12 +80,16 @@ plugin only if there is hard need to do so.**
    This option specifies whether imjournal should ignore messages
    currently in journal and read only new messages. This option is only
    used when there is no StateFile to avoid message loss.
+
 -  **DefaultSeverity** <severity>
+
    Some messages comming from journald don't have the SYSLOG_PRIORITY
    field. These are typically the messages logged through journald's
    native API. This option specifies the default severity for these
    messages. Can be given either as a name or a number. Defaults to 'notice'.
+
 -  **DefaultFacility** <facility>
+
    Some messages comming from journald don't have the SYSLOG_FACILITY
    field. These are typically the messages logged through journald's
    native API. This option specifies the default facility for these

source/configuration/modules/imptcp.rst

@@ -10,7 +10,7 @@ provide TLS services. Encryption can be provided by using
 This module has no limit on the number of listeners and sessions that
 can be used.
 
-**Author:**\ Rainer Gerhards <rgerhards@adiscon.com>
+**Author:** Rainer Gerhards <rgerhards@adiscon.com>
 
 Configuration Directives
 ------------------------
@@ -25,7 +25,8 @@ Module Parameters
 These paramters can be used with the "module()" statement. They apply
 globaly to all inputs defined by the module.
 
--  Threads <number>
+.. function:: Threads <number>
+
    Number of helper worker threads to process incoming messages. These
    threads are utilized to pull data off the network. On a busy system,
    additional helper threads (but not more than there are CPUs/Cores)
@@ -40,6 +41,31 @@ Input Parameters
 These parameters can be used with the "input()" statement. They apply to
 the input they are specified with.
 
+.. function:: port <number>
+
+   *Mandatory*
+
+   Select a port to listen on.
+
+.. function:: name <name>
+
+   Sets a name for the inputname property. If no name is set "imptcp"
+   is used by default. Setting a name is not strictly necessary, but can
+   be useful to apply filtering based on which input the message was
+   received from. Note that the name also shows up in
+   :doc:`impstats <impstats>` logs.
+
+.. function:: ruleset <name>
+
+   Binds specified ruleset to next server defined.
+
+.. function:: address <name>
+
+   *Default: all interfaces*
+
+   On multi-homed machines, specifies to which local address the
+   listerner should be bound.
+
 .. function:: AddtlFrameDelimiter <Delimiter>
 
    This directive permits to specify an additional frame delimiter for
@@ -116,26 +142,6 @@ the input they are specified with.
    This has only effect if keep-alive is enabled. The functionality may
    not be available on all platforms.
 
-.. function:: Port <number>
-
-   Select a port to listen on
-
-.. function:: Name <name>
-
-   Sets a name for the inputname property. If no name is set "imptcp"
-   is used by default. Setting a name is not strictly necessary, but can
-   be useful to apply filtering based on which input the message was
-   received from.
-
-.. function:: Ruleset <name>
-
-   Binds specified ruleset to next server defined.
-
-.. function:: Address <name>
-
-   On multi-homed machines, specifies to which local address the
-   listerner should be bound.
-
 .. function:: RateLimit.Interval [number]
 
    *Default is 0, which turns off rate limiting*
@@ -164,6 +170,14 @@ This sets up a TCP server on port 514:
   module(load="imptcp") # needs to be done just once 
   input(type="imptcp" port="514")
 
+This creates a listener that listens on the local loopback
+interface, only.
+
+::
+
+  module(load="imptcp") # needs to be done just once 
+  input(type="imptcp" port="514" address="127.0.0.1")
+
 Legacy Configuration Directives
 -------------------------------
 

source/configuration/modules/imtcp.rst

@@ -52,6 +52,31 @@ Global Parameters
    enable of disable keep-alive packets at the tcp socket layer. The
    default is to disable them.
 
+.. function:: KeepAlive.Probes <number>
+
+   The number of unacknowledged probes to send before considering the
+   connection dead and notifying the application layer. The default, 0,
+   means that the operating system defaults are used. This has only
+   effect if keep-alive is enabled. The functionality may not be
+   available on all platforms.
+
+.. function:: KeepAlive.Interval <number>
+
+   The interval between subsequential keepalive probes, regardless of
+   what the connection has exchanged in the meantime. The default, 0,
+   means that the operating system defaults are used. This has only
+   effect if keep-alive is enabled. The functionality may not be
+   available on all platforms.
+
+.. function:: KeepAlive.Time <number>
+
+   The interval between the last data packet sent (simple ACKs are not
+   considered data) and the first keepalive probe; after the connection
+   is marked to need keepalive, this counter is not used any further.
+   The default, 0, means that the operating system defaults are used.
+   This has only effect if keep-alive is enabled. The functionality may
+   not be available on all platforms.
+
 .. function:: FlowControl on/off
 
    *Default is on*
@@ -189,6 +214,18 @@ Legacy Configuration Directives
 
    equivalent to: KeepAlive
 
+.. function:: $InputTCPServerKeepAlive\_probes <number>
+
+   Equivalent to: KeepAlive.Probes
+
+.. function:: $InputTCPServerKeepAlive\_intvl <number>
+
+   Equivalent to: KeepAlive.Interval
+
+.. function:: $InputTCPServerKeepAlive\_time <number>
+
+   Equivalent to: KeepAlive.Time
+
 .. function:: $InputTCPServerRun <port>
 
    equivalent to: Port
@@ -235,6 +272,10 @@ Caveats/Known Bugs
 -  module always binds to all interfaces
 -  can not be loaded together with `imgssapi <imgssapi.html>`_ (which
    includes the functionality of imtcp)
+-  increasing MaxSessions and MaxListeners doesn't change MaxOpenFiles,
+   consider increasing this global configuration parameter too (on Linux
+   check the actual value for running process by in /proc/PID/limits; default
+   limit on Linux is 1024)
 
 Example
 -------

source/configuration/modules/imudp.rst

@@ -269,8 +269,8 @@ the OS until the user is sufficiently privileged.
 
 ::
 
-    module(load="imudp") # needs to be done just once input
-    (type="imudp" port="514" rcvbufSize="1m")
+    module(load="imudp") # needs to be done just once
+    input(type="imudp" port="514" rcvbufSize="1m")
 
 In the next example, we set up three listeners at ports 10514, 10515 and
 10516 and assign a listner name of "udp" to it, followed by the port

source/configuration/modules/mmfields.rst

@@ -72,18 +72,33 @@ Currently none.
 This is a very simple use case where each message is parsed. The default
 separator character of comma is being used.
 
-module(load="mmfields") template(name="ftpl" type=string
-string="%$!%\\n") action(type="omfields") action(type="omfile"
-file="/path/to/logfile" template="ftpl")
+:: 
+
+  module(load="mmfields")
+  template(name="ftpl"
+           type=string
+           string="%$!%\\n")
+  action(type="omfields")
+  action(type="omfile"
+         file="/path/to/logfile"
+         template="ftpl")
 
 The following sample is similar to the previous one, but this time the
 colon is used as separator and data is written into the "$!mmfields"
 json path.
 
-module(load="mmfields") template(name="ftpl" type=string
-string="%$!%\\n") action(type="omfields" separator=":"
-jsonRoot="!mmfields") action(type="omfile" file="/path/to/logfile"
-template="ftpl")
+::
+
+  module(load="mmfields")
+  template(name="ftpl"
+           type=string
+           string="%$!%\\n")
+  action(type="omfields"
+         separator=":"
+         jsonRoot="!mmfields")
+         action(type="omfile"
+         file="/path/to/logfile"
+         template="ftpl")
 
 This documentation is part of the `rsyslog <http://www.rsyslog.com/>`_
 project.

source/configuration/modules/mmjsonparse.rst

@@ -1,5 +1,5 @@
-Log Message Normalization Module
-================================
+JSON/CEE Structured Content Extraction Module (mmjsonparse)
+===========================================================
 
 **Module Name:    mmjsonparse**
 

source/configuration/modules/mmnormalize.rst

@@ -1,5 +1,5 @@
-Log Message Normalization Module
-================================
+Log Message Normalization Module (mmnormalize)
+==============================================
 
 **Module Name:    mmnormalize**
 
@@ -31,6 +31,14 @@ Note that mmnormalize should only be called once on each message.
 Behaviour is undefined if multiple calls to mmnormalize happen for the
 same message.
 
+**Module Parameters**
+
+-  **allow_regex** [boolean] defaults to "off"
+   Specifies if regex field-type should be allowed. Regex field-type has
+   significantly higher computational overhead compared to other fields, 
+   so it should be avoided when another field-type can achieve the desired 
+   effect. Needs to be "on" for regex field-type to work.
+
 **Action Parameters**:
 
 -  **ruleBase** [word]
@@ -49,6 +57,14 @@ same message.
    placed. By default, all parsed properties are merged into root of
    message properties. You can place them under a subtree, instead. You
    can place them in local variables, also, by setting path="$.".
+-  **variable** [word] *(Available since: 8.5.1)*
+   Specifies if a variable insteed of property 'msg' should be used for
+   normalization. A varible can be property, local variable, json-path etc.
+   Please note that **useRawMsg** overrides this parameter, so if **useRawMsg**
+   is set, **variable** will be ignored and raw message will be used.
+
+   
+
 
 **Legacy Configuration Directives**: