Skip to content

Commit 3ee0c71

Browse files
Update in_tail with group directive
Signed-off-by: Pranjal-Gupta2 <pranjal.gupta2@ibm.com>
1 parent 1e6f78f commit 3ee0c71

File tree

2 files changed

+106
-281
lines changed

2 files changed

+106
-281
lines changed

input/tail.md

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,16 @@ It is included in Fluentd's core.
1717
<parse>
1818
@type apache2
1919
</parse>
20+
<group>
21+
rate_period 30s
22+
<rule>
23+
match {
24+
namespace: /space1|space2|space2/,
25+
podname: /app.*/,
26+
}
27+
limit 200
28+
</rule>
29+
</group>
2030
</source>
2131
```
2232

@@ -402,6 +412,80 @@ The `@log_level` option allows the user to set different levels of logging for e
402412

403413
Refer to the [Logging](../deployment/logging.md) for more details.
404414

415+
416+
### `<group>` Section \(highly recommended\)
417+
418+
The `in_tail` plugin can assign each log file to a group, based on user defined rules. The `limit` parameter controls the total number of lines collected for a group within a `rate_period` time interval.
419+
420+
Example:
421+
422+
```text
423+
# group rules -- 1
424+
<group>
425+
rate_period 5s
426+
427+
<rule>
428+
match {
429+
namespace: /shopping/,
430+
podname: /frontend/,
431+
}
432+
limit 1000
433+
</rule>
434+
</group>
435+
436+
# group rules -- 2
437+
<group>
438+
<rule>
439+
match {
440+
directoy: /payment/
441+
}
442+
limit 2000
443+
</rule>
444+
</group>
445+
```
446+
447+
#### `pattern`
448+
449+
| type | default | version |
450+
| :--- | :--- | :--- |
451+
| regexp | `/^\/var\/log\/containers\/(?<podname>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\/[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace>[^_]+)_(?<container>.+)-(?<docker_id>[a-z0-9]{64})\.log$/`| 1.15 |
452+
453+
Specifies the regular expression for extracting metadata (namespace, podname) from log file path. Default value of the pattern regexp extracts information about `namespace`, `podname`, `docker_id`, `container` of the log (K8s specific).
454+
455+
You can also add custom named captures in `pattern` for custom grouping of log files. For example,
456+
```text
457+
pattern /^\/home\/logs\/(?<file>.+)\.log$/
458+
```
459+
In this example, filename will be extracted and used to form groups.
460+
461+
#### `rate_period`
462+
463+
| type | default | version |
464+
| :--- | :--- | :--- |
465+
| time | 60 \(seconds\) | 1.15 |
466+
467+
Time period in which the group line limit is applied. `in_tail` resets the counter after every `rate_period` interval.
468+
469+
#### `<rule>` Section \(required\)
470+
471+
Grouping rules for log files.
472+
473+
##### match
474+
475+
| type | default | version |
476+
| :--- | :--- | :--- |
477+
| hash | {"namespace": /./, "podname": /./} | 1.15 |
478+
479+
`match` parameter is used to check if a file belongs to a particular group based on hash keys (named captures from `pattern`) and hash values (regexp)
480+
481+
##### limit
482+
483+
| type | default | version |
484+
| :--- | :--- | :--- |
485+
| integer | -1 | 1.15 |
486+
487+
Maximum number of lines allowed from a group in `rate_period` time interval. The default value of `-1` doesn't throttle log files of that group.
488+
405489
## Learn More
406490

407491
* [Input Plugin Overview](./)
@@ -469,3 +553,25 @@ path C:\\path\\to\\*\\foo.log
469553

470554
If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.
471555

556+
### What happens when a file can be assigned to more than one group?
557+
558+
Example,
559+
560+
```text
561+
<rule> ## Rule1
562+
match {
563+
namespace: /monitoring/
564+
}
565+
limit 100
566+
</rule>
567+
568+
<rule> ## Rule2
569+
match {
570+
namespace: /monitoring/,
571+
podname: /logger/,
572+
}
573+
limit 2000
574+
</rule>
575+
```
576+
577+
In this case, rules with more constraints, i.e., greater number of `match` hash keys will be given a higher priority. So a file will be assigned to `Rule2` if it can be assigned to both `Rule1` and `Rule2`.

0 commit comments

Comments
 (0)