1
- 4.1.0
1
+ 4.1.1
2
2
=====
3
3
4
- ## Highlights
5
-
6
- ### PROXY protocol v2 support ([ #4211 ] ( https://github.com/syslog-ng/syslog-ng/pull/4211 ) )
7
-
8
- We've added support for PROXY protocol v2 (` transport(proxied-tcp) ` ), a protocol
9
- used by network load balancers, such as Amazon Elastic Load Balancer and
10
- HAProxy, to carry original source/destination address information, as described
11
- in https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
12
-
13
- ### Metrics revised
14
-
15
- #### Prometheus metric format ([ #4325 ] ( https://github.com/syslog-ng/syslog-ng/pull/4325 ) )
16
-
17
- A new metric system has been introduced to syslog-ng, where metrics are
18
- identified by names and partitioned by labels, which is similar to the
19
- [ Prometheus data model] ( https://prometheus.io/docs/concepts/data_model/ ) .
20
-
21
- The ` syslog-ng-ctl stats prometheus ` command can be used to query syslog-ng
22
- metrics in a format that conforms to the Prometheus text-based exposition
23
- format.
24
-
25
- ` syslog-ng-ctl stats prometheus --with-legacy-metrics ` displays legacy metrics
26
- as well. Legacy metrics do not follow Prometheus' metric and label conventions.
27
-
28
- #### Classification (metadata-based metrics) ([ #4318 ] ( https://github.com/syslog-ng/syslog-ng/pull/4318 ) )
29
-
30
- ` metrics-probe() ` , a new parser has also been added, which counts messages
31
- passing through based on the metadata of each message. The parser creates
32
- labeled metrics based on the fields of the message.
33
-
34
- Both the key and labels can be set in the config, the values of the labels can
35
- be templated. E.g.:
36
- ```
37
- parser p_metrics_probe {
38
- metrics-probe(
39
- key("custom_key") # adds "syslogng_" prefix => "syslogng_custom_key"
40
- labels(
41
- "custom_label_name_1" => "foobar"
42
- "custom_label_name_2" => "${.custom.field}"
43
- )
44
- );
45
- };
46
- ```
47
- With this config, it creates counters like these:
48
- ```
49
- syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="bar"} 1
50
- syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="foo"} 1
51
- syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="baz"} 3
52
- ```
53
-
54
- The minimal config creates counters with the key
55
- ` syslogng_classified_events_total ` and labels ` app ` , ` host ` , ` program ` and
56
- ` source ` . E.g.:
57
- ```
58
- parser p_metrics_probe {
59
- metrics-probe();
60
- };
61
- ```
62
- With this config, it creates counters like these:
63
- ```
64
- syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3
65
- syslogng_classified_events_total{app="example-app", host="localhost", program="bar", source="s_local_1"} 1
66
- syslogng_classified_events_total{app="example-app", host="localhost", program="foo", source="s_local_1"} 1
67
- ```
68
-
69
- #### Named log paths (path ingress/egress metrics) ([ #4344 ] ( https://github.com/syslog-ng/syslog-ng/pull/4344 ) )
70
-
71
- It is also possible to create named log paths, for example:
72
-
73
- ```
74
- log top-level {
75
- source(s_local);
76
-
77
- log inner-1 {
78
- filter(f_inner_1);
79
- destination(d_local_1);
80
- };
81
-
82
- log inner-2 {
83
- filter(f_inner_2);
84
- destination(d_local_2);
85
- };
86
- };
87
- ```
88
-
89
- Each named log path counts its ingress and egress messages:
90
- ```
91
- syslogng_log_path_ingress{id="top-level"} 114
92
- syslogng_log_path_ingress{id="inner-1"} 114
93
- syslogng_log_path_ingress{id="inner-2"} 114
94
- syslogng_log_path_egress{id="top-level"} 103
95
- syslogng_log_path_egress{id="inner-1"} 62
96
- syslogng_log_path_egress{id="inner-2"} 41
97
- ```
98
-
99
- Note that the egress statistics only count the messages which have been have not been filtered out from the related
100
- log path, it does care about whether there are any destinations in it or that any destination delivers or drops the
101
- message.
102
-
103
- The above three features are experimental; the output of ` stats prometheus `
104
- (names, labels, etc.) and the metrics created by ` metrics-probe() ` and named log
105
- paths may change in the next 2-3 releases.
106
-
107
- ## Features
108
-
109
- * ` $(format-date) ` : add a new template function to format time and date values
110
-
111
- ` $(format-date [options] format-string [timestamp]) `
112
-
113
- ` $(format-date) ` takes a timestamp in the DATETIME representation and
114
- formats it according to an strftime() format string. The DATETIME
115
- representation in syslog-ng is a UNIX timestamp formatted as a decimal
116
- number, with an optional fractional part, where the seconds and the
117
- fraction of seconds are separated by a dot.
118
-
119
- If the timestamp argument is missing, the timestamp of the message is
120
- used.
121
-
122
- Options:
123
- ` --time-zone <TZstring> ` -- override timezone of the original timestamp
124
- ([ #4202 ] ( https://github.com/syslog-ng/syslog-ng/pull/4202 ) )
125
-
126
- * ` syslog-parser() ` and all syslog related sources: accept unquoted RFC5424
127
- SD-PARAM-VALUEs instead of rejecting them with a parse error.
128
-
129
- ` sdata-parser() ` : this new parser allows you to parse an RFC5424 style
130
- structured data string. It can be used to parse this relatively complex
131
- format separately.
132
- ([ #4281 ] ( https://github.com/syslog-ng/syslog-ng/pull/4281 ) )
133
-
134
- * ` system() ` source: the ` system() ` source was changed on systemd platforms to
135
- fetch journal messages that relate to the current boot only (e.g. similar
136
- to ` journalctl -fb ` ) and to ignore messages generated in previous boots,
137
- even if those messages were succesfully stored in the journal and were not
138
- picked up by syslog-ng. This change was implemented as the journald access
139
- APIs work incorrectly if time goes backwards across reboots, which is an
140
- increasingly frequent event in virtualized environments and on systems that
141
- lack an RTC. If you want to retain the old behaviour, please bypass the
142
- ` system() ` source and use ` systemd-journal() ` directly, where this option
143
- can be customized. The change is not tied to ` @version ` as we deemed the new
144
- behaviour fixing an actual bug. For more information consult #2836 .
145
-
146
- ` systemd-journald() ` source: add ` match-boot() ` and ` matches() ` options to
147
- allow you to constrain the collection of journal records to a subset of what
148
- is in the journal. ` match-boot() ` is a yes/no value that allows you to fetch
149
- messages that only relate to the current boot. ` matches() ` allows you to
150
- specify one or more filters on journal fields.
151
-
152
- Examples:
153
-
154
- ```
155
- source s_journal_current_boot_only {
156
- systemd-source(match-boot(yes));
157
- };
158
-
159
- source s_journal_systemd_only {
160
- systemd-source(matches(
161
- "_COMM" => "systemd"
162
- )
163
- );
164
- };
165
- ```
166
- ([#4245](https://github.com/syslog-ng/syslog-ng/pull/4245))
167
-
168
- * `date-parser()`: add `value()` parameter to instruct `date-parser()` to store
169
- the resulting timestamp in a name-value pair, instead of changing the
170
- timestamp value of the LogMessage.
171
-
172
- `datetime` type representation: typed values in syslog-ng are represented as
173
- strings when stored as a part of a log message. syslog-ng simply remembers
174
- the type it was stored as. Whenever the value is used as a specific type in
175
- a type-aware context where we need the value of the specific type, an
176
- automatic string parsing takes place. This parsing happens for instance
177
- whenever syslog-ng stores a datetime value in MongoDB or when
178
- `$(format-date)` template function takes a name-value pair as parameter.
179
- The datetime() type has stored its value as the number of milliseconds since
180
- the epoch (1970-01-01 00:00:00 GMT). This has now been enhanced by making
181
- it possible to store timestamps up to nanosecond resolutions along with an
182
- optional timezone offset.
183
-
184
- `$(format-date)`: when applied to name-value pairs with the `datetime` type,
185
- use the timezone offset if one is available.
186
- ([#4319](https://github.com/syslog-ng/syslog-ng/pull/4319))
187
-
188
- * `stats`: Added `syslog-stats()` global `stats()` group option.
189
-
190
- E.g.:
191
- ```
192
- options {
193
- stats(
194
- syslog-stats(no);
195
- );
196
- };
197
- ```
198
-
199
- It changes the behavior of counting messages based on different syslog-proto fields,
200
- like `SEVERITY`, `FACILITY`, `HOST`, etc...
201
-
202
- Possible values are:
203
- * `yes` => force enable
204
- * `no` => force disable
205
- * `auto` => let `stats(level())` decide (old behavior)
206
- ([#4337](https://github.com/syslog-ng/syslog-ng/pull/4337))
207
-
208
- * `kubernetes` source: Added `key-delimiter()` option.
209
-
210
- Some metadata fields can contain `.`-s in their name. This does not work with syslog-ng-s macros, which
211
- by default use `.` as a delimiter. The added `key-delimiter()` option changes this behavior by storing
212
- the parsed metadata fields with a custom delimiter. In order to reach the fields, the accessor side has
213
- to use the new delimiter format, e.g. `--key-delimiter` option in `$(format-json)`.
214
- ([#4213](https://github.com/syslog-ng/syslog-ng/pull/4213))
215
-
216
4
## Bugfixes
217
5
218
- * Fix conditional evaluation with a dangling filter
219
-
220
- We've fixed a bug that caused conditional evaluation (if/else/elif) and certain logpath flags (`final`, `fallback`)
221
- to occasionally malfunction. The issue only happened in certain logpath constructs; examples can be found in the
222
- PR description.
223
- ([#4058](https://github.com/syslog-ng/syslog-ng/pull/4058))
224
- * `python`: Fixed a bug, where `PYTHONPATH` was ignored with `python3.11`.
225
- ([#4298](https://github.com/syslog-ng/syslog-ng/pull/4298))
226
- * `disk-buffer`: Fixed disk-queue file becoming corrupt when changing `disk-buf-size()`.
227
-
228
- `syslog-ng` now continues with the originally set `disk-buf-size()`.
229
- Note that changing the `disk-buf-size()` of an existing disk-queue was never supported,
230
- but could cause errors, which are fixed now.
231
- ([#4308](https://github.com/syslog-ng/syslog-ng/pull/4308))
232
- * `dqtool`: fix `dqtool assign`
233
- ([#4355](https://github.com/syslog-ng/syslog-ng/pull/4355))
234
- * `example-diskq-source`: Fixed failing to read the disk-queue content in some cases.
235
- ([#4308](https://github.com/syslog-ng/syslog-ng/pull/4308))
236
- * `default-network-drivers()`: Added support for the `log-iw-size()` option with a default value of 1000.
237
- Making it possible to adjust the `log-iw-size()` for the TCP/TLS based connections, when changing the `max-connections()` option.
238
- ([#4328](https://github.com/syslog-ng/syslog-ng/pull/4328))
239
- * `apache-accesslog-parser()`: fix rawrequest escaping binary characters
240
- ([#4303](https://github.com/syslog-ng/syslog-ng/pull/4303))
241
- * `dqtool`: Fixed `dqtool cat` failing to read the content in some cases.
242
- ([#4308](https://github.com/syslog-ng/syslog-ng/pull/4308))
243
- * Fixed a rare main loop related crash on FreeBSD.
244
- ([#4262](https://github.com/syslog-ng/syslog-ng/pull/4262))
245
- * Fix a warning message that was displayed incorrectly:
246
- "The actual number of worker threads exceeds the number of threads estimated at startup."
247
- ([#4282](https://github.com/syslog-ng/syslog-ng/pull/4282))
248
- * Fix minor memory leak related to tznames
249
- ([#4334](https://github.com/syslog-ng/syslog-ng/pull/4334))
250
-
251
- ## Packaging
252
-
253
- * `dbparser`: libdbparser.so has been renamed to libcorrelation.so.
254
- ([#4294](https://github.com/syslog-ng/syslog-ng/pull/4294))
255
- * `systemd-journal`: Fixed a linker error, which occurred, when building with `--with-systemd-journal=optional`.
256
- ([#4304](https://github.com/syslog-ng/syslog-ng/pull/4304))
257
- ([#4302](https://github.com/syslog-ng/syslog-ng/pull/4302))
258
-
259
- ## Notes to developers
260
-
261
- * `LogThreadedSourceDriver` and `Fetcher`: implement source-side batching
262
- support on the input path by assigning a thread_id to dynamically spawned
263
- input threads (e.g. those spawned by LogThreadedSourceDriver) too. To
264
- actually improve performance the source driver should disable automatic
265
- closing of batches by setting `auto_close_batches` to FALSE and calling
266
- log_threaded_source_close_batch() explicitly.
267
- ([#3969](https://github.com/syslog-ng/syslog-ng/pull/3969))
268
-
269
- ## Other changes
270
-
271
- * stats related options: The stats related options have been groupped to a new `stats()` block.
272
-
273
- This affects the following global options:
274
- * `stats-freq()`
275
- * `stats-level()`
276
- * `stats-lifetime()`
277
- * `stats-max-dynamics()`
278
-
279
- These options have been kept for backward compatibility, but they have been deprecated.
280
-
281
- Migrating from the old stats options to the new ones looks like this.
282
- ```
283
- @version: 4.0
284
-
285
- options {
286
- stats-freq(1);
287
- stats-level(1);
288
- stats-lifetime(1000);
289
- stats-max-dynamics(10000);
290
- };
291
- ```
292
- ```
293
- @version: 4.1
294
-
295
- options {
296
- stats(
297
- freq(1)
298
- level(1)
299
- lifetime(1000)
300
- max-dynamics(10000)
301
- );
302
- };
303
- ```
304
-
305
- **Breaking change**
306
- For more than a decade `stats()` was a deprecated alias to `stats-freq()`, now it is used as the name
307
- of the new block. If you have been using `stats(xy)`, use `stats(freq(xy))` instead.
308
- ([#4337](https://github.com/syslog-ng/syslog-ng/pull/4337))
309
- * `kubernetes` source: Improved error logging, when the pod was unreachable through the python API.
310
- ([#4305](https://github.com/syslog-ng/syslog-ng/pull/4305))
311
- * APT repository: Added .gz, .xz and .bz2 compression to the Packages file.
312
- ([#4313](https://github.com/syslog-ng/syslog-ng/pull/4313))
6
+ * ` db-parser() ` , ` grouping-by() ` : Fixed a crash introduced in 4.1.0.
7
+ ([ #4366 ] ( https://github.com/syslog-ng/syslog-ng/pull/4366 ) )
313
8
314
9
## Credits
315
10
@@ -322,5 +17,6 @@ of syslog-ng, contribute.
322
17
323
18
We would like to thank the following people for their contribution:
324
19
325
- Attila Szakacs, Balazs Scheidler, Bálint Horváth, Gergo Ferenc Kovacs,
326
- Hofi, László Várady, Ronny Meeus, Szilard Parrag
20
+ Attila Szakacs, Balazs Scheidler, Kyle Brown
21
+
22
+
0 commit comments