Skip to content

Commit 47d7af5

Browse files
kenhyskenhys
authored
Merge pull request #527 from daipom/add-source-only-mode
source-only mode: add initial document
2 parents 2483910 + 6197f3f commit 47d7af5

File tree

7 files changed

+275
-0
lines changed

7 files changed

+275
-0
lines changed

‎.gitbook/assets/source-only-mode-mechanism.png

182 KB
Loading

‎SUMMARY.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@
5656
* [Fluentd UI](deployment/fluentd-ui.md)
5757
* [Linux Capability](deployment/linux-capability.md)
5858
* [Command Line Option](deployment/command-line-option.md)
59+
* [Source Only Mode](deployment/source-only-mode.md)
5960
* [Container Deployment](container-deployment/README.md)
6061
* [Docker Image](container-deployment/install-by-docker.md)
6162
* [Docker Logging Driver](container-deployment/docker-logging-driver.md)
@@ -97,6 +98,7 @@
9798
* [mongo\_replset](output/mongo_replset.md)
9899
* [rewrite\_tag\_filter](output/rewrite_tag_filter.md)
99100
* [webhdfs](output/webhdfs.md)
101+
* [buffer](output/buffer.md)
100102
* [Filter Plugins](filter/README.md)
101103
* [record\_transformer](filter/record_transformer.md)
102104
* [grep](filter/grep.md)

‎deployment/command-line-option.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ Usage: fluentd [options]
3333
--suppress-repeated-stacktrace [VALUE]
3434
suppress repeated stacktrace
3535
--without-source invoke a fluentd without input plugins
36+
--with-source-only Invoke a fluentd only with input plugins. The data is stored in a temporary buffer. Send SIGWINCH to cancel this mode and process the data (Not supported on Windows).
3637
--use-v1-config Use v1 configuration format (default)
3738
--use-v0-config Use v0 configuration format
3839
--strict-config-value Parse config values strictly
@@ -53,6 +54,7 @@ Usage: fluentd [options]
5354
* `--suppress-config-dump`: Fluentd starts without configuration dump. If you do not want to show the configuration in fluentd logs, e.g. it contains private keys, then this option is useful.
5455
* `--suppress-repeated-stacktrace`: If `true`, suppresses the stacktrace in fluentd logs. Since v0.12, this option is `true` by default.
5556
* `--without-source`: Fluentd starts without input plugins. This option is useful for flushing buffers with no new incoming events.
57+
* `--with-source-only` (Not supported on Windows): See [Source Only Mode](source-only-mode.md) for details.
5658
* `-i`, `--inline-config`: If fluentd is used on XaaS which does not support persistent disks, this option is useful.
5759
* `--no-supervisor`: If you want to use your supervisor tools, this option avoids double supervisor.
5860

‎deployment/signals.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,4 +32,8 @@ If you use fluentd v1.9.0 or later, use `SIGUSR2` instead.
3232

3333
Calls SIGDUMP to dump fluentd internal status. See [troubleshooting](trouble-shooting.md#dump-fluentds-internal-information) article.
3434

35+
### SIGWINCH
36+
37+
Cancels [Source Only Mode](source-only-mode.md).
38+
3539
If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.

‎deployment/source-only-mode.md

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Source Only Mode
2+
3+
Since v1.18.0, Fluentd can launch with source-only mode.
4+
(Not supported on Windows)
5+
6+
* In this mode, only input plugins run.
7+
* During this mode, the events are stored in a temporary file buffer.
8+
* Sending `SIGWINCH` to the supervisor cancels this mode.
9+
* Then, all plugins start to run, and the temporary file buffer starts to load.
10+
11+
## How to launch Fluentd with source-only mode
12+
13+
You can launch Fluentd with source-only mode in the following ways.
14+
15+
* [Command Line Option - --with-source-only](command-line-option.md)
16+
* [System Configuration - with_source_only](system-config.md#with_source_only)
17+
18+
## Temporary file buffer
19+
20+
During source-only mode, the ingested events are stored in a temporary file buffer.
21+
After `SIGWINCH` is sent to the supervisor and this mode is canceled, this buffer begins to load.
22+
23+
By default, the file buffer path is as follows.
24+
25+
* `/tmp/fluent/source-only-buffer/{Unique ID for the Fluentd instance}`
26+
27+
You can confirm it in the following log output at startup.
28+
29+
```
30+
[info]: #0 with-source-only: the emitted data will be stored in the buffer files under
31+
/tmp/fluent/source-only-buffer/bbd9006d-bc41-418b-b346-f80888641dda. You can send SIGWINCH to
32+
the supervisor process to cancel with-source-only mode and process data.
33+
```
34+
35+
This file buffer is the buffer of [out_buffer](../output/buffer.md).
36+
It works with the default settings of `out_buffer` except for the following points:
37+
38+
* `path` is automatically determined by default.
39+
* `overflow_action` is `drop_oldest_chunk` by default.
40+
41+
If needed, you can configure some options for the buffer in System Configuration.
42+
Please see the following for details.
43+
44+
* [System Configuration - source_only_buffer section](system-config.md#less-than-source_only_buffer-greater-than-section).
45+
46+
### Recovery
47+
48+
If Fluentd stops with the temporary buffer remained, you need to recover the buffer to launch Fluentd with source-only mode again.
49+
50+
Note that a different path will be used each time unless you configure the temporary buffer path explicitly.
51+
In this case, you can recover the buffer as follows.
52+
53+
1. Configure the remaining buffer path explicitly.
54+
1. Start Fluentd with source-only mode again.
55+
1. Send `SIGWINCH` to the supervisor to load the buffer.
56+
57+
If this recovery is necessary, i.e., Fluentd stops with the temporary buffer remained, the following warning log will be displayed.
58+
You can confirm the path to configure by this log.
59+
60+
```
61+
[warn]: #0 some buffer files remain in /tmp/fluent/source-only-buffer/bbd9006d-bc41-418b-b346-f80888641dda.
62+
Please consider recovering or saving the buffer files in the directory. To recover them, you can set
63+
the buffer path manually to system config and retry, i.e., restart Fluentd with with-source-only mode
64+
and send SIGWINCH again. Config Example:
65+
<system>
66+
<source_only_buffer>
67+
path /tmp/fluent/source-only-buffer/bbd9006d-bc41-418b-b346-f80888641dda
68+
</source_only_buffer>
69+
</system>
70+
```
71+
72+
## Mechanism
73+
74+
![source-only-mode mechanism](../.gitbook/assets/source-only-mode-mechanism.png)
75+
76+
If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.

‎deployment/system-config.md

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,20 @@ Logs event verbosely.
100100

101101
Invokes fluentd without input plugins.
102102

103+
### `with_source_only`
104+
105+
| type | default | version |
106+
| :--- | :--- | :--- |
107+
| bool | nil | 1.18.0 |
108+
109+
Not supported on Windows.
110+
111+
Invokes a fluentd only with input plugins.
112+
113+
See [Source Only Mode](source-only-mode.md) for details.
114+
115+
See also [source_only_buffer section](#less-than-source_only_buffer-greater-than-section).
116+
103117
### `rpc_endpoint`
104118

105119
| type | default | version |
@@ -245,6 +259,68 @@ Specifies whether measuring input metrics should be enabled or not.
245259
Specifies whether measuring record size metrics should be enabled or not.
246260
It can be useful for calculating flow rate on Fluentd instances.
247261

262+
### `<source_only_buffer>` section
263+
264+
The temporary buffer setting for [Source Only Mode](source-only-mode.md).
265+
266+
#### `flush_thread_count`
267+
268+
| type | default | version |
269+
| :--- | :--- | :--- |
270+
| integer | 1 | 1.18.0 |
271+
272+
See [Output Plugins - flush_thread_count](../output#flush_thread_count) for details.
273+
274+
#### `overflow_action`
275+
276+
| type | default | version |
277+
| :--- | :--- | :--- |
278+
| enum | drop_oldest_chunk | 1.18.0 |
279+
280+
See [Output Plugins - overflow_action](../output#overflow_action) for details.
281+
282+
#### `path`
283+
284+
| type | default | version |
285+
| :--- | :--- | :--- |
286+
| string | nil (use unique path for the instance) | 1.18.0 |
287+
288+
Set this option when recovering buffers ([Source Only Mode - Recovery](source-only-mode.md#Recovery)).
289+
290+
See [Buffer Plugins - file - path](../buffer/file.md#path) for details.
291+
292+
#### `flush_interval`
293+
294+
| type | default | version |
295+
| :--- | :--- | :--- |
296+
| integer | 10 | 1.18.0 |
297+
298+
See [Output Plugins - flush_interval](../output#flush_interval) for details.
299+
300+
#### `chunk_limit_size`
301+
302+
| type | default | version |
303+
| :--- | :--- | :--- |
304+
| size | 256MB | 1.18.0 |
305+
306+
See [Buffer Section Configurations - Buffering Parameters](../configuration/buffer-section.md#buffering-parameters) for details.
307+
308+
#### `total_limit_size`
309+
310+
| type | default | version |
311+
| :--- | :--- | :--- |
312+
| size | 64GB | 1.18.0 |
313+
314+
See [Buffer Section Configurations - Buffering Parameters](../configuration/buffer-section.md#buffering-parameters) for details.
315+
316+
#### `compress`
317+
318+
| type | default | version |
319+
| :--- | :--- | :--- |
320+
| enum | text | 1.18.0 |
321+
322+
See [Buffer Section Configurations - Buffering Parameters](../configuration/buffer-section.md#buffering-parameters) for details.
323+
248324
### `Fluent::SystemConfig::Mixin` Methods
249325

250326
#### `.system_config`
@@ -285,6 +361,7 @@ For code example, please refer to [`api-plugin-base`](../plugin-development/api-
285361
* `--emit-error-log-interval SECONDS`: Suppresses the interval \(seconds\) to emit error logs.
286362
* `--suppress-repeated-stacktrace [VALUE]`: Suppress the repeated stacktrace.
287363
* `--without-source`: Invokes fluentd without input plugins.
364+
* `--with-source-only` (Not supported on Windows): Invokes a fluentd only with input plugins. See [Source Only Mode](source-only-mode.md) for details.
288365
* `--use-v1-config`: Uses v1 configuration format \(default\).
289366
* `--use-v0-config`: Uses v0 configuration format.
290367
* `--strict-config-value`: Parses the configuration values strictly. Invalid numerical or boolean values are rejected. Fluentd raises configuration error exceptions instead of replacing them with `0` or `true` implicitly.

‎output/buffer.md

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# buffer
2+
3+
The `buffer` output plugin buffers and re-labels events.
4+
This plugin is similar to [out_relabel](relabel.md), but uses [buffer](../buffer/).
5+
6+
It is included in Fluentd's core (since v1.18.0).
7+
8+
## Example Configuration
9+
10+
```
11+
<source>
12+
@type udp
13+
@label @buffer
14+
tag foo.udp
15+
<parse>
16+
@type none
17+
</parse>
18+
</source>
19+
20+
<label @buffer>
21+
<match **>
22+
@type buffer
23+
@label @ROOT
24+
<buffer>
25+
path /path/to/buffer
26+
</buffer>
27+
</match>
28+
</label>
29+
30+
<match foo.**>
31+
@type stdout
32+
</match>
33+
```
34+
35+
In the above example, events ingested by `in_udp` are once stored in the buffer of this plugin, then re-routed and output by `out_stdout`.
36+
37+
## Supported Modes
38+
39+
* Synchronous Buffered
40+
41+
See also: [Output Plugin Overview](./)
42+
43+
## Plugin Helpers
44+
45+
* [`event_emitter`](../plugin-helper-overview/api-plugin-helper-event_emitter.md)
46+
47+
## Parameters
48+
49+
[Common Parameters](../configuration/plugin-common-parameters.md)
50+
51+
### `@type` \(required\)
52+
53+
The value must be `buffer`.
54+
55+
### `@label` \(required\)
56+
57+
| type | default | version |
58+
| :--- | :--- | :--- |
59+
| string | `nil` | 1.18.0 |
60+
61+
Specifies the label to re-route.
62+
63+
Note: You can specify `@ROOT` to re-route to the root.
64+
65+
### `<buffer>` Section
66+
67+
#### `path` \(required\)
68+
69+
| type | default | version |
70+
| :--- | :--- | :--- |
71+
| string | required parameter | 1.18.0 |
72+
73+
#### `@type`
74+
75+
| type | default | version |
76+
| :--- | :--- | :--- |
77+
| string | file | 1.18.0 |
78+
79+
Overwrites the default value in this plugin.
80+
81+
#### `chunk_keys`
82+
83+
| type | default | version |
84+
| :--- | :--- | :--- |
85+
| array | tag | 1.18.0 |
86+
87+
Overwrites the default value in this plugin.
88+
89+
#### `flush_mode`
90+
91+
| type | default | version |
92+
| :--- | :--- | :--- |
93+
| enum | interval | 1.18.0 |
94+
95+
Overwrites the default value in this plugin.
96+
97+
#### `flush_interval`
98+
99+
| type | default | version |
100+
| :--- | :--- | :--- |
101+
| integer | 10 | 1.18.0 |
102+
103+
Overwrites the default value in this plugin.
104+
105+
#### Common Buffer / Output parameters
106+
107+
In addition, you can configure other common settings.
108+
Please see the followings for details.
109+
110+
* [Buffer Section Configurations](../configuration/buffer-section.md)
111+
* [Buffer Plugin Overview](../buffer/)
112+
* [Output Plugin Overview](./)
113+
114+
If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.

0 commit comments

Comments
 (0)