You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pipeline/filters/kubernetes.md
+21-5Lines changed: 21 additions & 5 deletions
Original file line number
Diff line number
Diff line change
@@ -13,6 +13,8 @@ When Fluent Bit is deployed in Kubernetes as a DaemonSet and configured to read
13
13
* Pod ID
14
14
* Labels
15
15
* Annotations
16
+
* Namespace Labels
17
+
* Namespace Annotations
16
18
17
19
The data is cached locally in memory and appended to each record.
18
20
@@ -40,18 +42,21 @@ The plugin supports the following configuration parameters:
40
42
| Regex\_Parser | Set an alternative Parser to process record Tag and extract pod\_name, namespace\_name, container\_name and docker\_id. The parser must be registered in a [parsers file](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf)\(refer to parser _filter-kube-test_ as an example\). ||
41
43
| K8S-Logging.Parser | Allow Kubernetes Pods to suggest a pre-defined Parser \(read more about it in Kubernetes Annotations section\)| Off |
42
44
| K8S-Logging.Exclude | Allow Kubernetes Pods to exclude their logs from the log processor \(read more about it in Kubernetes Annotations section\). | Off |
43
-
| Labels | Include Kubernetes resource labels in the extra metadata. | On |
44
-
| Annotations | Include Kubernetes resource annotations in the extra metadata. | On |
45
+
| Labels | Include Kubernetes pod resource labels in the extra metadata. | On |
46
+
| Annotations | Include Kubernetes pod resource annotations in the extra metadata. | On |
45
47
| Kube\_meta\_preload\_cache\_dir | If set, Kubernetes meta-data can be cached/pre-loaded from files in JSON format in this directory, named as namespace-pod.meta ||
46
48
| Dummy\_Meta | If set, use dummy-meta data \(for test/dev purposes\)| Off |
47
49
| DNS\_Retries | DNS lookup retries N times until the network start working | 6 |
48
50
| DNS\_Wait\_Time | DNS lookup interval between network status checks | 30 |
49
-
| Use\_Kubelet | this is an optional feature flag to get metadata information from kubelet instead of calling Kube Server API to enhance the log. This could mitigate the [Kube API heavy traffic issue for large cluster](kubernetes.md#optional-feature-using-kubelet-to-get-metadata). | Off |
51
+
| Use\_Kubelet | this is an optional feature flag to get metadata information from kubelet instead of calling Kube Server API to enhance the log. This could mitigate the [Kube API heavy traffic issue for large cluster](kubernetes.md#optional-feature-using-kubelet-to-get-metadata). If used when any [Kubernetes Namespace Meta](#kubernetes-namespace-meta) fields are enabled, Kubelet will be used to fetch pod data, but namespace meta will still be fetched using the `Kube_URL` settings.| Off |
50
52
| Kubelet\_Port | kubelet port using for HTTP request, this only works when `Use_Kubelet` set to On. | 10250 |
51
53
| Kubelet\_Host | kubelet host using for HTTP request, this only works when `Use_Kubelet` set to On. | 127.0.0.1 |
52
-
| Kube\_Meta\_Cache\_TTL | configurable TTL for K8s cached metadata. By default, it is set to 0 which means TTL for cache entries is disabled and cache entries are evicted at random when capacity is reached. In order to enable this option, you should set the number to a time interval. For example, set this value to 60 or 60s and cache entries which have been created more than 60s will be evicted. | 0 |
54
+
| Kube\_Meta\_Cache\_TTL | configurable TTL for K8s cached pod metadata. By default, it is set to 0 which means TTL for cache entries is disabled and cache entries are evicted at random when capacity is reached. In order to enable this option, you should set the number to a time interval. For example, set this value to 60 or 60s and cache entries which have been created more than 60s will be evicted. | 0 |
53
55
| Kube\_Token\_TTL | configurable 'time to live' for the K8s token. By default, it is set to 600 seconds. After this time, the token is reloaded from Kube_Token_File or the Kube_Token_Command.| 600 |
54
56
| Kube\_Token\_Command | Command to get Kubernetes authorization token. By default, it will be `NULL` and we will use token file to get token. If you want to manually choose a command to get it, you can set the command here. For example, run `aws-iam-authenticator -i your-cluster-name token --token-only` to set token. This option is currently Linux-only. ||
57
+
| Kube\_Meta\_Namespace\_Cache\_TTL | configurable TTL for K8s cached namespace metadata. By default, it is set to 900 which means a 15min TTL for namespace cache entries. Setting this to 0 will mean entries are evicted at random once the cache is full. | 900 |
58
+
| Namespace\_Labels | Include Kubernetes namespace resource labels in the extra metadata. See [Kubernetes Namespace Meta](#kubernetes-namespace-meta)| Off |
59
+
| Namespace\_Annotations | Include Kubernetes namespace resource annotations in the extra metadata. See [Kubernetes Namespace Meta](#kubernetes-namespace-meta)| Off |
55
60
56
61
## Processing the 'log' value
57
62
@@ -76,7 +81,18 @@ To perform processing of the _log_ key, it's **mandatory to enable** the _Merge\
76
81
77
82
If _log_ value processing fails, the value is untouched. The order above is not chained, meaning it's exclusive and the filter will try only one of the options above, **not** all of them.
78
83
79
-
## Kubernetes Annotations
84
+
## Kubernetes Namespace Meta
85
+
Namespace Meta can be enabled via the following settings:
86
+
* Namespace\_Labels
87
+
* Namespace\_Annotations
88
+
89
+
Using any Namespace Meta requires the use of the Kube API as it can not be fetched directly from Kubelet. If `Use_Kubelet On` has been set, the Kubelet api will only be used to fetch pod metadata, while namespace meta is fetched from the upstream Kubernetes API.
90
+
91
+
Namespace Meta if collected will be stored within a `kubernetes_namespace` record key.
92
+
93
+
> Namespace meta is not be guaranteed to be in sync as namespace labels & annotations can be adjusted after pod creation. Adjust `Kube_Meta_Namespace_Cache_TTL` to lower caching times to fit your use case.
94
+
95
+
## Kubernetes Pod Annotations
80
96
81
97
A flexible feature of Fluent Bit Kubernetes filter is that allow Kubernetes Pods to suggest certain behaviors for the log processor pipeline when processing the records. At the moment it support:
0 commit comments