source-only mode: add initial document

SUMMARY.md

@@ -56,6 +56,7 @@
   * [Fluentd UI](deployment/fluentd-ui.md)
   * [Linux Capability](deployment/linux-capability.md)
   * [Command Line Option](deployment/command-line-option.md)
+  * [Source Only Mode](deployment/source-only-mode.md)
 * [Container Deployment](container-deployment/README.md)
   * [Docker Image](container-deployment/install-by-docker.md)
   * [Docker Logging Driver](container-deployment/docker-logging-driver.md)
@@ -97,6 +98,7 @@
   * [mongo\_replset](output/mongo_replset.md)
   * [rewrite\_tag\_filter](output/rewrite_tag_filter.md)
   * [webhdfs](output/webhdfs.md)
+  * [buffer](output/buffer.md)
 * [Filter Plugins](filter/README.md)
   * [record\_transformer](filter/record_transformer.md)
   * [grep](filter/grep.md)

deployment/command-line-option.md

@@ -33,6 +33,7 @@ Usage: fluentd [options]
         --suppress-repeated-stacktrace [VALUE]
                                      suppress repeated stacktrace
         --without-source             invoke a fluentd without input plugins
+        --with-source-only           Invoke a fluentd only with input plugins. The data is stored in a temporary buffer. Send SIGWINCH to cancel this mode and process the data (Not supported on Windows).
         --use-v1-config              Use v1 configuration format (default)
         --use-v0-config              Use v0 configuration format
         --strict-config-value        Parse config values strictly
@@ -53,6 +54,7 @@ Usage: fluentd [options]
 * `--suppress-config-dump`: Fluentd starts without configuration dump. If you do not want to show the configuration in fluentd logs, e.g. it contains private keys, then this option is useful.
 * `--suppress-repeated-stacktrace`: If `true`, suppresses the stacktrace in fluentd logs. Since v0.12, this option is `true` by default.
 * `--without-source`: Fluentd starts without input plugins. This option is useful for flushing buffers with no new incoming events.
+* `--with-source-only` (Not supported on Windows): See [Source Only Mode](source-only-mode.md) for details.
 * `-i`, `--inline-config`: If fluentd is used on XaaS which does not support persistent disks, this option is useful.
 * `--no-supervisor`: If you want to use your supervisor tools, this option avoids double supervisor.
 

deployment/signals.md

@@ -32,4 +32,8 @@ If you use fluentd v1.9.0 or later, use `SIGUSR2` instead.
 
 Calls SIGDUMP to dump fluentd internal status. See [troubleshooting](trouble-shooting.md#dump-fluentds-internal-information) article.
 
+### SIGWINCH
+
+Cancels [Source Only Mode](source-only-mode.md).
+
 If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.

deployment/source-only-mode.md

@@ -0,0 +1,76 @@
+# Source Only Mode
+
+Since v1.18.0, Fluentd can launch with source-only mode.
+(Not supported on Windows)
+
+* In this mode, only input plugins run.
+* During this mode, the events are stored in a temporary file buffer.
+* Sending `SIGWINCH` to the supervisor cancels this mode.
+  * Then, all plugins start to run, and the temporary file buffer starts to load.
+
+## How to launch Fluentd with source-only mode
+
+You can launch Fluentd with source-only mode in the following ways.
+
+* [Command Line Option - --with-source-only](command-line-option.md)
+* [System Configuration - with_source_only](system-config.md#with_source_only)
+
+## Temporary file buffer
+
+During source-only mode, the ingested events are stored in a temporary file buffer.
+After `SIGWINCH` is sent to the supervisor and this mode is canceled, this buffer begins to load.
+
+By default, the file buffer path is as follows.
+
+* `/tmp/fluent/source-only-buffer/{Unique ID for the Fluentd instance}`
+
+You can confirm it in the following log output at startup.
+
+```
+[info]: #0 with-source-only: the emitted data will be stored in the buffer files under
+/tmp/fluent/source-only-buffer/bbd9006d-bc41-418b-b346-f80888641dda. You can send SIGWINCH to
+the supervisor process to cancel with-source-only mode and process data.
+```
+
+This file buffer is the buffer of [out_buffer](../output/buffer.md).
+It works with the default settings of `out_buffer` except for the following points:
+
+* `path` is automatically determined by default.
+* `overflow_action` is `drop_oldest_chunk` by default.
+
+If needed, you can configure some options for the buffer in System Configuration.
+Please see the following for details.
+
+* [System Configuration - source_only_buffer section](system-config.md#less-than-source_only_buffer-greater-than-section).
+
+### Recovery
+
+If Fluentd stops with the temporary buffer remained, you need to recover the buffer to launch Fluentd with source-only mode again.
+
+Note that a different path will be used each time unless you configure the temporary buffer path explicitly.
+In this case, you can recover the buffer as follows.
+
+1. Configure the remaining buffer path explicitly.
+1. Start Fluentd with source-only mode again.
+1. Send `SIGWINCH` to the supervisor to load the buffer.
+
+If this recovery is necessary, i.e., Fluentd stops with the temporary buffer remained, the following warning log will be displayed.
+You can confirm the path to configure by this log.
+
+```
+[warn]: #0 some buffer files remain in /tmp/fluent/source-only-buffer/bbd9006d-bc41-418b-b346-f80888641dda.
+Please consider recovering or saving the buffer files in the directory. To recover them, you can set
+the buffer path manually to system config and retry, i.e., restart Fluentd with with-source-only mode
+and send SIGWINCH again. Config Example:
+<system>
+  <source_only_buffer>
+    path /tmp/fluent/source-only-buffer/bbd9006d-bc41-418b-b346-f80888641dda
+  </source_only_buffer>
+</system>
+```
+
+## Mechanism
+
+![source-only-mode mechanism](../.gitbook/assets/source-only-mode-mechanism.png)
+
+If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.

deployment/system-config.md

@@ -100,6 +100,20 @@ Logs event verbosely.
 
 Invokes fluentd without input plugins.
 
+### `with_source_only`
+
+| type | default | version |
+| :--- | :--- | :--- |
+| bool | nil | 1.18.0 |
+
+Not supported on Windows.
+
+Invokes a fluentd only with input plugins.
+
+See [Source Only Mode](source-only-mode.md) for details.
+
+See also [source_only_buffer section](#less-than-source_only_buffer-greater-than-section).
+
 ### `rpc_endpoint`
 
 | type | default | version |
@@ -245,6 +259,68 @@ Specifies whether measuring input metrics should be enabled or not.
 Specifies whether measuring record size metrics should be enabled or not.
 It can be useful for calculating flow rate on Fluentd instances.
 
+### `<source_only_buffer>` section
+
+The temporary buffer setting for [Source Only Mode](source-only-mode.md).
+
+#### `flush_thread_count`
+
+| type    | default | version |
+| :---    | :---    | :---    |
+| integer | 1       | 1.18.0  |
+
+See [Output Plugins - flush_thread_count](../output#flush_thread_count) for details.
+
+#### `overflow_action`
+
+| type | default           | version |
+| :--- | :---              | :---    |
+| enum | drop_oldest_chunk | 1.18.0  |
+
+See [Output Plugins - overflow_action](../output#overflow_action) for details.
+
+#### `path`
+
+| type   | default                                | version |
+| :---   | :---                                   | :---    |
+| string | nil (use unique path for the instance) | 1.18.0  |
+
+Set this option when recovering buffers ([Source Only Mode - Recovery](source-only-mode.md#Recovery)).
+
+See [Buffer Plugins - file - path](../buffer/file.md#path) for details.
+
+#### `flush_interval`
+
+| type    | default | version |
+| :---    | :---    | :---    |
+| integer | 10      | 1.18.0  |
+
+See [Output Plugins - flush_interval](../output#flush_interval) for details.
+
+#### `chunk_limit_size`
+
+| type | default | version |
+| :--- | :---    | :---    |
+| size | 256MB   | 1.18.0  |
+
+See [Buffer Section Configurations - Buffering Parameters](../configuration/buffer-section.md#buffering-parameters) for details.
+
+#### `total_limit_size`
+
+| type | default | version |
+| :--- | :---    | :---    |
+| size | 64GB    | 1.18.0  |
+
+See [Buffer Section Configurations - Buffering Parameters](../configuration/buffer-section.md#buffering-parameters) for details.
+
+#### `compress`
+
+| type | default | version |
+| :--- | :---    | :---    |
+| enum | text    | 1.18.0  |
+
+See [Buffer Section Configurations - Buffering Parameters](../configuration/buffer-section.md#buffering-parameters) for details.
+
 ### `Fluent::SystemConfig::Mixin` Methods
 
 #### `.system_config`
@@ -285,6 +361,7 @@ For code example, please refer to [`api-plugin-base`](../plugin-development/api-
 * `--emit-error-log-interval SECONDS`: Suppresses the interval \(seconds\) to emit error logs.
 * `--suppress-repeated-stacktrace [VALUE]`: Suppress the repeated stacktrace.
 * `--without-source`: Invokes fluentd without input plugins.
+* `--with-source-only` (Not supported on Windows): Invokes a fluentd only with input plugins. See [Source Only Mode](source-only-mode.md) for details.
 * `--use-v1-config`: Uses v1 configuration format \(default\).
 * `--use-v0-config`: Uses v0 configuration format.
 * `--strict-config-value`: Parses the configuration values strictly. Invalid numerical or boolean values are rejected. Fluentd raises configuration error exceptions instead of replacing them with `0` or `true` implicitly.

output/buffer.md

@@ -0,0 +1,114 @@
+# buffer
+
+The `buffer` output plugin buffers and re-labels events.
+This plugin is similar to [out_relabel](relabel.md), but uses [buffer](../buffer/).
+
+It is included in Fluentd's core (since v1.18.0).
+
+## Example Configuration
+
+```
+<source>
+  @type udp
+  @label @buffer
+  tag foo.udp
+  <parse>
+    @type none
+  </parse>
+</source>
+
+<label @buffer>
+  <match **>
+    @type buffer
+    @label @ROOT
+    <buffer>
+      path /path/to/buffer
+    </buffer>
+  </match>
+</label>
+
+<match foo.**>
+  @type stdout
+</match>
+```
+
+In the above example, events ingested by `in_udp` are once stored in the buffer of this plugin, then re-routed and output by `out_stdout`.
+
+## Supported Modes
+
+* Synchronous Buffered
+
+See also: [Output Plugin Overview](./)
+
+## Plugin Helpers
+
+* [`event_emitter`](../plugin-helper-overview/api-plugin-helper-event_emitter.md)
+
+## Parameters
+
+[Common Parameters](../configuration/plugin-common-parameters.md)
+
+### `@type` \(required\)
+
+The value must be `buffer`.
+
+### `@label` \(required\)
+
+| type   | default | version |
+| :---   | :---    | :---    |
+| string | `nil`   | 1.18.0  |
+
+Specifies the label to re-route.
+
+Note: You can specify `@ROOT` to re-route to the root.
+
+### `<buffer>` Section
+
+#### `path` \(required\)
+
+| type   | default            | version |
+| :---   | :---               | :---    |
+| string | required parameter | 1.18.0  |
+
+#### `@type`
+
+| type   | default | version |
+| :---   | :---    | :---    |
+| string | file    | 1.18.0  |
+
+Overwrites the default value in this plugin.
+
+#### `chunk_keys`
+
+| type  | default | version |
+| :---  | :---    | :---    |
+| array | tag     | 1.18.0  |
+
+Overwrites the default value in this plugin.
+
+#### `flush_mode`
+
+| type | default  | version |
+| :--- | :---     | :---    |
+| enum | interval | 1.18.0  |
+
+Overwrites the default value in this plugin.
+
+#### `flush_interval`
+
+| type    | default | version |
+| :---    | :---    | :---    |
+| integer | 10      | 1.18.0  |
+
+Overwrites the default value in this plugin.
+
+#### Common Buffer / Output parameters
+
+In addition, you can configure other common settings.
+Please see the followings for details.
+
+* [Buffer Section Configurations](../configuration/buffer-section.md)
+* [Buffer Plugin Overview](../buffer/)
+* [Output Plugin Overview](./)
+
+If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.