Skip to content

Commit f0aba67

Browse files
Watson1978Watson1978
authored
container-deployment: update install-by-docker.md (#563)
* container-deployment: update install-by-docker.md Signed-off-by: Shizuo Fujita <fujita@clear-code.com> * container-deployment: add a way how to use plugins directory Signed-off-by: Shizuo Fujita <fujita@clear-code.com> * container-deployment: remove exclamation mark Signed-off-by: Shizuo Fujita <fujita@clear-code.com> --------- Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
1 parent 29a6f6e commit f0aba67

File tree

1 file changed

+107
-27
lines changed

1 file changed

+107
-27
lines changed

‎container-deployment/docker-logging-driver.md

Lines changed: 107 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,6 @@ The old-fashioned way is to write these messages into a log file, but that inher
66

77
On Docker v1.6, the concept of [**logging drivers**](https://docs.docker.com/engine/admin/logging/overview/) was introduced. The Docker engine is aware of the output interfaces that manage the application messages.
88

9-
![Fluentd Docker](https://www.fluentd.org/assets/img/recipes/fluentd_docker.png)
10-
119
For Docker v1.8, we have implemented a native [**Fluentd Docker logging driver**](https://docs.docker.com/engine/admin/logging/fluentd/). Now, you are able to have a unified and structured logging system with the simplicity and high performance of [Fluentd](http://fluentd.org).
1210

1311
NOTE: Currently, the Fluentd logging driver doesn't support sub-second precision.
@@ -38,7 +36,7 @@ Create `demo.conf` with the following configuration:
3836
bind 0.0.0.0
3937
</source>
4038
41-
<match *>
39+
<match **>
4240
@type stdout
4341
</match>
4442
```
@@ -48,14 +46,16 @@ Create `demo.conf` with the following configuration:
4846
Now, start an instance of Fluentd like this:
4947

5048
```text
51-
$ docker run -it -p 24224:24224 -v $(pwd)/demo.conf:/fluentd/etc/demo.conf -e FLUENTD_CONF=demo.conf fluent/fluentd:latest
49+
$ docker run -it -p 24224:24224 -v $(pwd)/demo.conf:/fluentd/etc/demo.conf -e FLUENTD_CONF=demo.conf fluent/fluentd:edge-debian
5250
```
5351

5452
On successful start, you should see the Fluentd startup logs:
5553

5654
```text
57-
2019-08-21 00:51:02 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/demo.conf"
58-
2019-08-21 00:51:02 +0000 [info]: using configuration file: <ROOT>
55+
2025-02-04 07:52:25 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
56+
2025-02-04 07:52:25 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/demo.conf"
57+
2025-02-04 07:52:26 +0000 [info]: gem 'fluentd' version '1.16.7'
58+
2025-02-04 07:52:26 +0000 [info]: using configuration file: <ROOT>
5959
<source>
6060
@type forward
6161
port 24224
@@ -65,27 +65,25 @@ On successful start, you should see the Fluentd startup logs:
6565
@type stdout
6666
</match>
6767
</ROOT>
68-
2019-08-21 00:51:02 +0000 [info]: starting fluentd-1.3.2 pid=6 ruby="2.5.2"
69-
2019-08-21 00:51:02 +0000 [info]: spawn command to main: cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/demo.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
70-
2019-08-21 00:51:02 +0000 [info]: gem 'fluentd' version '1.3.2'
71-
2019-08-21 00:51:02 +0000 [info]: adding match pattern="*" type="stdout"
72-
2019-08-21 00:51:02 +0000 [info]: adding source type="forward"
73-
2019-08-21 00:51:02 +0000 [info]: #0 starting fluentd worker pid=16 ppid=6 worker=0
74-
2019-08-21 00:51:02 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
75-
2019-08-21 00:51:02 +0000 [info]: #0 fluentd worker is now running worker=0
68+
2025-02-04 07:52:26 +0000 [info]: starting fluentd-1.16.7 pid=7 ruby="3.2.6"
69+
2025-02-04 07:52:26 +0000 [info]: spawn command to main: cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "--config", "/fluentd/etc/demo.conf", "--plugin", "/fluentd/plugins", "--under-supervisor"]
70+
2025-02-04 07:52:26 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
71+
2025-02-04 07:52:26 +0000 [info]: adding match pattern="*" type="stdout"
72+
2025-02-04 07:52:26 +0000 [info]: adding source type="forward"
73+
2025-02-04 07:52:26 +0000 [info]: #0 starting fluentd worker pid=16 ppid=7 worker=0
74+
2025-02-04 07:52:26 +0000 [info]: #0 listening port port=24224 bind="0.0.0.0"
75+
2025-02-04 07:52:26 +0000 [info]: #0 fluentd worker is now running worker=0
7676
```
7777

7878
### Step 3: Start Docker Container with Fluentd Driver
7979

8080
By default, the Fluentd logging driver will try to find a local Fluentd instance \(Step \# 2\) listening for connections on the TCP port `24224`. Note that the container will not start if it cannot connect to the Fluentd instance.
8181

82-
![Integration of Docker with Fluentd](https://www.fluentd.org/assets/img/recipes/fluentd_docker_integrated.png)
83-
8482
The following command will run a base Ubuntu container and print some messages to the standard output:
8583

8684
```text
87-
$ docker run --log-driver=fluentd ubuntu echo "Hello Fluentd!"
88-
Hello Fluentd!
85+
$ docker run --log-driver=fluentd ubuntu echo "Hello Fluentd"
86+
Hello Fluentd
8987
```
9088

9189
Note that we have launched the container specifying the Fluentd logging driver i.e. `--log-driver=fluentd`.
@@ -95,7 +93,7 @@ Note that we have launched the container specifying the Fluentd logging driver i
9593
Now, you should see the incoming messages from the container in Fluentd logs:
9694

9795
```text
98-
2019-08-21 00:52:28.000000000 +0000 ece4524df531: {"source":"stdout","log":"Hello Fluentd!","container_id":"ece4524df531ed6ded4253c145a53bead9b049241aa12c5a59ab83e3a14a96b4","container_name":"/inspiring_montalcini"}
96+
2025-02-04 07:52:46.000000000 +0000 a4289c14a0ba: {"container_id":"a4289c14a0ba4716deff4d2aadc2b9a51331c0b9a5d631115060ffda959b2bc3","container_name":"/vigilant_babbage","source":"stdout","log":"Hello Fluentd"}
9997
```
10098

10199
At this point, you will notice that the incoming messages are in JSON format, have a timestamp, are tagged with the `container_id` and contain general information from the source container along with the message.
@@ -105,6 +103,7 @@ At this point, you will notice that the incoming messages are in JSON format, ha
105103
The application log is stored in the `"log"` field in the record. You can parse this log before sending it to the destinations by using [`filter_parser`](../filter/parser.md).
106104

107105
```text
106+
# filter configuration
108107
<filter docker.**>
109108
@type parser
110109
key_name log
@@ -113,50 +112,131 @@ The application log is stored in the `"log"` field in the record. You can parse
113112
@type json # apache2, nginx, etc.
114113
</parse>
115114
</filter>
115+
116+
<source>
117+
@type forward
118+
port 24224
119+
bind 0.0.0.0
120+
</source>
121+
122+
<match **>
123+
@type stdout
124+
</match>
125+
```
126+
127+
Then you provide the log message with JSON format:
128+
129+
```text
130+
$ docker run --log-driver=fluentd --log-opt tag=docker ubuntu echo "{\"key\":\"value\"}"
116131
```
117132

118-
Original Event:
133+
About `--log-opt tag=...`, please refer at [Driver Options](#driver-options) section.
134+
135+
Original Event (without filter plugin):
119136

120137
```text
121-
2019-07-22 03:36:39.000000000 +0000 6e8a14315069: {"log":"{\"key\":\"value\"}","container_id":"6e8a1431506936b8568a284f2b0dd4853c250ad85ab7a497f05c4d371f6c3ae6","container_name":"/laughing_beaver","source":"stdout"}
138+
2025-02-04 08:20:07.000000000 +0000 docker: {"container_id":"291757f94709abd945ba07c7769c46a5c011f014119c0c56b79e287b00dad4ab","container_name":"/awesome_euler","source":"stdout","log":"{\"key\":\"value\"}"}
122139
```
123140

124141
Filtered Event:
125142

126143
```text
127-
2019-07-22 03:35:59.395952500 +0000 bac5426337a6: {"container_id":"bac5426337a611fc3b7a0b318c3c45981d2acd80f5c5651088bebb8f1f962583","container_name":"/nostalgic_euler","source":"stdout","log":"{\"key\":\"value\"}","key":"value"}
144+
2025-02-04 08:20:37.969077831 +0000 docker: {"container_id":"291757f94709abd945ba07c7769c46a5c011f014119c0c56b79e287b00dad4ab","container_name":"/awesome_euler","source":"stdout","log":"{\"key\":\"value\"}","key":"value"}
128145
```
129146

130147
### Additional Step 2: Concatenate Multiple Lines Log Messages
131148

132149
The application log is stored in the `log` field of the record. You can concatenate these logs by using [`fluent-plugin-concat`](https://github.com/fluent-plugins-nursery/fluent-plugin-concat) filter before sending it to the destinations.
133150

151+
At first, you need to create custom docker image due to install the `fluent-plugin-concat` gem in the Fluentd container.
152+
153+
Create `Dockerfile` with the following content:
154+
155+
```text
156+
# Dockerfile
157+
FROM fluent/fluentd:edge-debian
158+
159+
USER root
160+
RUN fluent-gem install fluent-plugin-concat
161+
162+
USER fluent
163+
```
164+
165+
Build the custom image:
166+
167+
```text
168+
$ docker build . -t fluentd-test
169+
```
170+
171+
Then, create the configuration file `demo.conf` with the following content:
172+
134173
```text
174+
# filter configuration
135175
<filter docker.**>
136176
@type concat
137177
key log
138178
stream_identity_key container_id
139179
multiline_start_regexp /^-e:2:in `\/'/
140180
multiline_end_regexp /^-e:4:in/
141181
</filter>
182+
183+
<source>
184+
@type forward
185+
port 24224
186+
bind 0.0.0.0
187+
</source>
188+
189+
<match **>
190+
@type stdout
191+
</match>
142192
```
143193

144-
Original Events:
194+
Launch the Fluentd container:
195+
196+
```
197+
$ docker run -it -p 24224:24224 -v $(pwd)/demo.conf:/fluentd/etc/demo.conf -e FLUENTD_CONF=demo.conf fluentd-test
198+
```
199+
200+
Then you provide the log message contains newlines:
145201

146202
```text
147-
2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky","source":"stdout","log":"-e:2:in `/'"}
148-
2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"source":"stdout","log":"-e:2:in `do_division_by_zero'","container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky"}
149-
2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"source":"stdout","log":"-e:4:in `<main>'","container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky"}
203+
$ docker run --log-driver=fluentd --log-opt tag=docker ubuntu echo "-e:2:in \`/'"$'\n'"-e:2:in \`do_division_by_zero'"$'\n'"-e:4:in \`<main>'"
204+
```
205+
206+
Original Events (without filter plugin):
207+
208+
```text
209+
2025-02-04 09:18:12.000000000 +0000 docker: {"container_id":"6998df1c3ad699d40abb58ab9f5cd4cd4ee51fb1bf99389deed64c2ffe439418","container_name":"/vigilant_ganguly","source":"stdout","log":"-e:2:in `/'"}
210+
2025-02-04 09:18:12.000000000 +0000 docker: {"container_id":"6998df1c3ad699d40abb58ab9f5cd4cd4ee51fb1bf99389deed64c2ffe439418","container_name":"/vigilant_ganguly","source":"stdout","log":"-e:2:in `do_division_by_zero'"}
211+
2025-02-04 09:18:12.000000000 +0000 docker: {"container_id":"6998df1c3ad699d40abb58ab9f5cd4cd4ee51fb1bf99389deed64c2ffe439418","container_name":"/vigilant_ganguly","source":"stdout","log":"-e:4:in `<main>'"}
150212
```
151213

152214
Filtered Events:
153215

154216
```text
155-
2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky","source":"stdout","log":"-e:2:in `/'\n-e:2:in `do_division_by_zero'\n-e:4:in `<main>'"}
217+
2025-02-04 09:18:30.000000000 +0000 docker: {"container_id":"6998df1c3ad699d40abb58ab9f5cd4cd4ee51fb1bf99389deed64c2ffe439418","container_name":"/vigilant_ganguly","source":"stdout","log":"-e:2:in `/'\n-e:2:in `do_division_by_zero'\n-e:4:in `<main>'"}
156218
```
157219

158220
If the logs are typical stacktraces, consider using [`detect-exceptions`](https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions) plugin instead.
159221

222+
**NOTE**:
223+
For plugins with the simple file structure, such as `fluent-plugin-concat`, `plugins` directory can be used instead of creating custom docker image.
224+
225+
Prepare the `plugins` directory and copy the plugin file:
226+
227+
```text
228+
$ mkdir $(pwd)/plugins
229+
230+
$ git clone https://github.com/fluent-plugins-nursery/fluent-plugin-concat.git /tmp/fluent-plugin-concat
231+
$ cp /tmp/fluent-plugin-concat/lib/fluent/plugin/filter_concat.rb $(pwd)/plugins
232+
```
233+
234+
Launch the Fluentd container with `plugins` directory mounted:
235+
236+
```
237+
$ docker run -it -p 24224:24224 -v $(pwd)/plugins:/fluentd/plugins -v $(pwd)/demo.conf:/fluentd/etc/demo.conf -e FLUENTD_CONF=demo.conf fluent/fluentd:edge-debian
238+
```
239+
160240
## Driver Options
161241

162242
The [Fluentd Logging Driver](https://docs.docker.com/engine/admin/logging/fluentd/) supports following options through the `--log-opt` Docker command-line argument:

0 commit comments

Comments
 (0)