Wildcard file monitoring fixes/changes

_data/navigation.yml

@@ -793,7 +793,7 @@ admin-guide-nav:
       subnav:
       - title: "Options of CSV parsers"
         url: /admin-guide/120_Parser/004_CSV_parser/000_CSV_parser_options
-    - title: "Parsing dates and timestamps"
+    - title: "Dates and timestamps parsing"
       url: /admin-guide/120_Parser/005_dates_parser/README
       subnav:
       - title: "Options of date-parser() parsers"
@@ -908,16 +908,16 @@ admin-guide-nav:
         url: /admin-guide/120_Parser/012_Linux_audit_parser/000_Linux_audit_parser_options
     - title: "MariaDB parser"
       url: /admin-guide/120_Parser/013_mariadb_parser
-    - title: "metrics-probe"
+    - title: "metrics probe"
       url: /admin-guide/120_Parser/014_metrics_probe_parser/README
       subnav:
       - title: "metrics-probe options"
         url: /admin-guide/120_Parser/014_metrics_probe_parser/000_metrics_probe_options
-    - title: "Parsing tags"
+    - title: "Tags parser"
       url: /admin-guide/120_Parser/014_tags_parser
     - title: "Netskope parser"
       url: /admin-guide/120_Parser/015_netskope_parser
-    - title: "panos-parser"
+    - title: "panos parser"
       url: /admin-guide/120_Parser/015_panos_parser/README
       subnav:
       - title: "Message format parsed by panos-parser()"

_includes/doc/admin-guide/options/follow-freq.md

@@ -3,12 +3,16 @@
 | Type:| number|
 |Default:  | 1|
 
-*Description:* Indicates that the source should be checked periodically.
-This is useful for files which always indicate readability, even though
-no new content were appended (e.g. regular file system files). If this value
-is higher than zero, syslog-ng will not attempt to use kqueue or ivykis file change
-notification methods on the file (poll(), epoll(), etc.), but checks whether
-the file changed every time the follow-freq() interval (in seconds) has elapsed.\
+*Description:* This is useful for files that always indicate readability, even though
+no new content has been appended (e.g., regular file system files).\
+In pre-4.9 versions of {{ site.product.short_name }}, where follow-method() was not yet available,
+this indicated that the source should be checked periodically. If this value is higher than zero,
+{{ site.product.short_name }} will not attempt to use kqueue or ivykis file change notification
+methods on the file (poll(), epoll(), etc.), but instead checks whether the file has changed every time the follow-freq()
+interval (in seconds) has elapsed.\
+In version 4.9 and later, this option only has an effect if follow-method() is set to `legacy` or `poll`,
+ in which case it is used as in pre-4.9 versions. For any other follow-method() values, the value of this option is ignored.
+
 Floating-point numbers (for example, **1.5**) can be used as well.
 
 {% include doc/admin-guide/warnings/file-source-follow-warning.md %}

_includes/doc/admin-guide/options/follow-method.md

@@ -0,0 +1,15 @@
+## follow-method()
+
+|Accepted values:| legacy \| inotify \| poll \| system |
+|Default: | legacy |
+
+*Description:* This option is available in version 4.9 and higher and its value controls how {{ site.product.short_name }} will follow file changes.\
+The default `legacy` mode preserves the pre-4.9 version file follow-mode behavior of {{ site.product.short_name }}, which is based on the value of follow-freq().\
+The `poll` value forces {{ site.product.short_name }} to poll for file changes at the interval specified by the monitor-freq() option, even if a more efficient method (such as `inotify` or `kqueue`) is available.\
+If `inotify` is selected and supported by the platform, {{ site.product.short_name }} uses it to detect changes in source files. This is the most efficient and least resource-consuming option available on Linux for regular files.\
+The `system` value will use system poll methods (via ivykis) like `port-timer` `port` `dev_poll` `epoll-timerfd` `epoll` `kqueue` `ppoll` `poll` and `uring`. For more information about how to control the system polling methods used, see How content changes are followed in file() and wildcard-file() sources.
+
+**NOTE:** Using `inotify` also requires setting monitor-method() to `inotify`.
+{: .notice--info}
+
+{% include doc/admin-guide/warnings/file-source-follow-warning.md %}

_includes/doc/admin-guide/warnings/file-source-follow-warning.md

@@ -1,2 +1,2 @@
-![]({{ site.baseurl}}/assets/images/caution.png) **CAUTION:** As the follow-freq(), the monitor-method(), and the monitor-freq() options could have performance penalty effects, it is important to understand how the file and directory monitoring operates depending on these values. For details, refer to How content changes are followed in file() and wildcard-file() sources.
+![]({{ site.baseurl}}/assets/images/caution.png) **CAUTION:** As the follow-freq(), follow-method(), monitor-method(), and the monitor-freq() options could have performance penalty effects, it is important to understand how the file and directory monitoring operates depending on these values. For details, refer to How content changes are followed in file() and wildcard-file() sources.
 {: .notice--warning}

doc/_admin-guide/060_Sources/020_File/000_File_source_options.md

@@ -17,6 +17,8 @@ The file() driver has the following options:
 
 {% include doc/admin-guide/options/follow-freq.md %}
 
+{% include doc/admin-guide/options/follow-method.md %}
+
 {% include doc/admin-guide/options/hook.md %}
 
 {% include doc/admin-guide/options/keep-timestamp.md %}

doc/_admin-guide/060_Sources/020_File/001_File_following.md

@@ -19,16 +19,22 @@ Setting the `auto` method will select the best available method on the given OS.
 
 Detecting file content changes involves more factors that can affect resource usage and overall performance. {{ site.product.short_name }} uses two main methods to monitor file content changes.
 
-The first method is automatically selected if the follow-freq() option has a value greater than 0. It works like the directory monitoring `poll` monitor-method() and uses an (ivykis) timer with the frequency of follow-freq(). It tries to detect changes in the file content (as well as state, file deletion, and moves) each time the timer fires. Similar to directory change monitoring, this process can be resource-intensive, so you should find the proper setting to balance performance, resource usage, and fault tolerance (such as avoiding log message loss).
+The first method is automatically selected in pre-4.9 version of {{ site.product.short_name }} if the follow-freq() option has a value greater than 0, or in version 4.9 or higher if follow-method() `legacy` or `poll` selected. It works like the directory monitoring `poll` monitor-method() and uses an (ivykis) timer with the frequency of follow-freq(). It tries to detect changes in the file content (as well as state, file deletion, and moves) each time the timer fires.\
+Similar to directory change monitoring, this process can be resource-intensive, so you should find the proper setting to balance performance, resource usage, and fault tolerance (such as avoiding log message loss).
 
-The second method is activated if the follow-freq() option is set to 0. It uses ivykis poll methods, sometimes resembling the poll method for directory change watching described above (with its performance penalties), but often working similarly to the `inotify` or `kqueue` version (with seamless performance).
+The second method is activated in pre-4.9 versions of {{ site.product.short_name }} if the follow-freq() option is set to 0, or in version 4.9 and higher if follow-method() is set to inotify or system.\
+If inotify is available on the system and selected in both follow-method() and monitor-method(), it will be used, resulting in significantly lower resource usage (especially on Linux). This option is accessible only on version 4.9 or later.\
+Otherwise, {{ site.product.short_name }} uses ivykis polling methods, which sometimes resemble the polling method used for directory change watching described above (with its associated performance penalties), but can also operate similarly to the kqueue version (offering seamless performance).
 
 The following table shows which method is selected in different cases.
 
 <table border="0" cellspacing="0" cellpadding="0" width="1397">
   <tr>
-    <td width="99" class="right-edged-col">
-      <p align="center">follow-freq()</p>
+    <td width="174" class="right-edged-col">
+      <p align="center">pre-4.9 version<br>or<br>follow-method(legacy)<br>or<br>follow-method(poll)</p>
+    </td>
+    <td width="151" class="right-edged-col">
+      <p align="center">version 4.9 or higher<br>follow-method()</p>
     </td>
     <td width="200" colspan="2" class="right-edged-col">
       <p align="center">file follow method</p>
@@ -44,8 +50,11 @@ The following table shows which method is selected in different cases.
     </td>
   </tr>
   <tr>
-    <td width="99" rowspan="8" class="right-edged-col">
-      <p align="center">0</p>
+    <td width="174" rowspan="8" class="right-edged-col">
+      <p align="center">follow-freq(0)</p>
+    </td>
+    <td width="150" rowspan="7" class="right-edged-col">
+      <p align="center">system</p>
     </td>
     <td width="150" rowspan="7" class="right-edged-col">
       <p align="center">ivykis poll</p>
@@ -148,7 +157,10 @@ The following table shows which method is selected in different cases.
     </td>
   </tr>
   <tr>
-    <td width="150"  colspan="2" class="right-edged-col">
+    <td width="150" class="right-edged-col">
+      <p align="center">inotify</p>
+    </td>
+    <td width="150" colspan="2" class="right-edged-col">
       <p align="center">inotify from ivykis directory monitor</p>
     </td>
     <td width="151" class="right-edged-col">
@@ -162,8 +174,11 @@ The following table shows which method is selected in different cases.
     </td>
   </tr>
   <tr>
-    <td width="99" rowspan="3" class="right-edged-col">
-      <p align="center">&gt; 0</p>
+    <td width="174" rowspan="3" class="right-edged-col">
+      <p align="center">follow-freq(n)<br> n &gt; 0</p>
+    </td>
+    <td width="150" rowspan="3" class="right-edged-col">
+      <p align="center">poll</p>
     </td>
     <td width="150" colspan="2" rowspan="3" class="right-edged-col">
       <p align="center">syslog-ng poll<br>using an ivykis timer with timer freq set to follow-freq() value</p>
@@ -180,7 +195,7 @@ The following table shows which method is selected in different cases.
   </tr>
 </table>
 
-  **NOTE:** As you can see, the best-performing option on Linux is the `inotify from ivykis directory monitor` method, which requires inotify kernel support, monitor-method() set to `inotify` or `auto` and follow-freq() set to 0.
+  **NOTE:** As you can see, the best-performing option on Linux is the `inotify from ivykis directory monitor` method, which requires inotify kernel support, monitor-method() set to `inotify` or `auto` and follow-method() set to `inotify`.
   {: .notice--info}
 
 A bit more detail about the notation in the platform columns and what they really mean:

doc/_admin-guide/060_Sources/030_Wildcard-file/000_Wildcard-file_options.md

@@ -76,6 +76,8 @@ source s_files {
 
 {% include doc/admin-guide/options/follow-freq.md %}
 
+{% include doc/admin-guide/options/follow-method.md %}
+
 {% include doc/admin-guide/options/hook.md %}
 
 {% include doc/admin-guide/options/keep-timestamp.md %}

doc/_admin-guide/060_Sources/155_stdin/000_stdin_options.md

@@ -18,6 +18,8 @@ The stdin() driver has the following options:
 
 {% include doc/admin-guide/options/follow-freq.md %}
 
+{% include doc/admin-guide/options/follow-method.md %}
+
 {% include doc/admin-guide/options/hook.md %}
 
 {% include doc/admin-guide/options/keep-timestamp.md %}