out_pgsql: add a configurable value called "daemon" for out pgsql plugin #7215

TomlinfreeGit · 2023-04-18T06:09:32Z

update out_pgsql plugin: add a configurable parameter to support run this plugin in a daemon mode.
if in daemon mode, configuration error of out_pgsql will not cause fluentbit crash or exit.

N/A

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Run local packaging test showing all targets (including any new ones) build.
Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

Backporting

Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

TomlinfreeGit · 2023-04-18T06:15:17Z

example config file for running out_pgsql in daemon mode:

[SERVICE]
    Flush  5
    Daemon  False
    Log_Level  error

[INPUT]
    name  cpu
    tag  cpu.local

[OUTPUT]
    name  pgsql
    host  192.168.142.132
    port  15432
    user  postgres
    password  yourpassword
    match  *
    database  agent
    table  test
    Daemon  True

[OUTPUT]
    name  stdout
    match  *

running result and valgrind result:

example config file for running out_pgsql not in daemon mode:

[SERVICE]
    Flush  5
    Daemon  False
    Log_Level  error

[INPUT]
    name  cpu
    tag  cpu.local

[OUTPUT]
    name  pgsql
    host  192.168.142.132
    port  15432
    user  postgres
    password  yourpassword
    match  *
    database  agent
    table  test
    Daemon  False

[OUTPUT]
    name  stdout
    match  *

running result and valgrind result:

TomlinfreeGit · 2023-04-18T06:28:56Z

related document Merge Request
fluent/fluent-bit-docs#1081

edsiper · 2023-04-25T09:23:35Z

@sxd pls take a look

sxd · 2023-04-25T09:59:00Z

@edsiper will take a look later today

@TomlinfreeGit the DCO test it's failing, can you please fix that in the meantime?

Best Regards!

plugins/out_pgsql/pgsql.c

sxd · 2023-04-25T19:39:00Z

plugins/out_pgsql/pgsql_connections.c

@@ -143,6 +143,10 @@ int pgsql_next_connection(struct flb_pgsql_config *ctx)
    struct mk_list *head;
    int ret_conn = 1;

+    if (ctx ==NULL) {


another space it's missing here too

fixed in the commit: out_pgsql: code-format &if config err then log err

sxd · 2023-04-25T19:44:14Z

@TomlinfreeGit so, the idea it's the just restart or make the error not a crash here right? what if the user don't see this error ? how they will suppose to know that there's an error in the configuration ?

Signed-off-by: Tom <yao.lin@siemens.com>

TomlinfreeGit · 2023-04-26T02:05:06Z

@TomlinfreeGit so, the idea it's the just restart or make the error not a crash here right? what if the user don't see this error ? how they will suppose to know that there's an error in the configuration ?

Yes, for other out_plugins, error configuration will not cause fluent-bit crash, just log error when failed to flush data, and can be noticed by health_check; However for out_pgsql, error configuration will cause fluent-bit crash directly.

the purpose for this PR is to make it possible out_pgsql deal with error configuration the same way as other out_plugins.
there will be error log both for init and flush process for out_pgsql configuration errors, if user don't see the logs, they can also notice this by health_check.

plugins/out_pgsql/pgsql.c

sxd · 2023-04-26T10:56:20Z

@TomlinfreeGit Testing this after a couple of minutes, about 10 minutes, I don't see any error in the logs saying that something needs to be fixed, the plugin just finished, doesn't look like a desire behavior since everything will work and will start, but if I can't flush the logs to the desired database this may be an issue.

This it's what I can see in the logs:

Fluent Bit v2.1.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/04/26 12:50:03] [error] [output:pgsql:pgsql.0] failed connecting to host=192.168.2.69 with error: connection to server at "192.168.2.69", port 5432 failed: No route to host
	Is the server running on that host and accepting TCP/IP connections?

And that's it, now on the API I'm getting this:

curl -i http://127.0.0.1:2020/api/v2/health
HTTP/1.1 200 OK
Server: Monkey/1.7.0
Date: Wed, 26 Apr 2023 10:53:51 GMT
Transfer-Encoding: chunked
Content-Type: application/json

{"fluent-bit":{"version":"2.1.0","edition":"Community","flags":["FLB_HAVE_IN_STORAGE_BACKLOG","FLB_HAVE_CHUNK_TRACE","FLB_HAVE_PARSER","FLB_HAVE_RECORD_ACCESSOR","FLB_HAVE_STREAM_PROCESSOR","FLB_HAVE_TLS","FLB_HAVE_OPENSSL","FLB_HAVE_METRICS","FLB_HAVE_WASM","FLB_HAVE_AWS","FLB_HAVE_AWS_CREDENTIAL_PROCESS","FLB_HAVE_SIGNV4","FLB_HAVE_SQLDB","FLB_HAVE_METRICS","FLB_HAVE_HTTP_SERVER","FLB_HAVE_SYSTEMD","FLB_HAVE_VALGRIND","FLB_HAVE_FORK","FLB_HAVE_TIMESPEC_GET","FLB_HAVE_GMTOFF","FLB_HAVE_UNIX_SOCKET","FLB_HAVE_ATTRIBUTE_ALLOC_SIZE","FLB_HAVE_PROXY_GO","FLB_HAVE_LIBBACKTRACE","FLB_HAVE_REGEX","FLB_HAVE_UTF8_ENCODER","FLB_HAVE_LUAJIT","FLB_HAVE_C_TLS","FLB_HAVE_ACCEPT4","FLB_HAVE_INOTIFY","FLB_HAVE_GETENTROPY","FLB_HAVE_GETENTROPY_SYS_RANDOM"]}}

So, how someone will notice that there's an error if everything start without problem? in my opinion a daemon should fail if there's anything that may cause an error, just like inside a pod/container, it will be restarted until something it's fixed, this it's more like making an error kind of invisible.

On the other hand, which plugins in Fluent-Bit behaves like what you want to add?
Can you elaborate more on why we should implement this? I may be wrong so I want to read more

Thanks in advance!

TomlinfreeGit · 2023-04-27T01:47:10Z

@TomlinfreeGit Testing this after a couple of minutes, about 10 minutes, I don't see any error in the logs saying that something needs to be fixed, the plugin just finished, doesn't look like a desire behavior since everything will work and will start, but if I can't flush the logs to the desired database this may be an issue.

This it's what I can see in the logs:
Fluent Bit v2.1.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/04/26 12:50:03] [error] [output:pgsql:pgsql.0] failed connecting to host=192.168.2.69 with error: connection to server at "192.168.2.69", port 5432 failed: No route to host
	Is the server running on that host and accepting TCP/IP connections?
And that's it, now on the API I'm getting this:
curl -i http://127.0.0.1:2020/api/v2/health
HTTP/1.1 200 OK
Server: Monkey/1.7.0
Date: Wed, 26 Apr 2023 10:53:51 GMT
Transfer-Encoding: chunked
Content-Type: application/json

{"fluent-bit":{"version":"2.1.0","edition":"Community","flags":["FLB_HAVE_IN_STORAGE_BACKLOG","FLB_HAVE_CHUNK_TRACE","FLB_HAVE_PARSER","FLB_HAVE_RECORD_ACCESSOR","FLB_HAVE_STREAM_PROCESSOR","FLB_HAVE_TLS","FLB_HAVE_OPENSSL","FLB_HAVE_METRICS","FLB_HAVE_WASM","FLB_HAVE_AWS","FLB_HAVE_AWS_CREDENTIAL_PROCESS","FLB_HAVE_SIGNV4","FLB_HAVE_SQLDB","FLB_HAVE_METRICS","FLB_HAVE_HTTP_SERVER","FLB_HAVE_SYSTEMD","FLB_HAVE_VALGRIND","FLB_HAVE_FORK","FLB_HAVE_TIMESPEC_GET","FLB_HAVE_GMTOFF","FLB_HAVE_UNIX_SOCKET","FLB_HAVE_ATTRIBUTE_ALLOC_SIZE","FLB_HAVE_PROXY_GO","FLB_HAVE_LIBBACKTRACE","FLB_HAVE_REGEX","FLB_HAVE_UTF8_ENCODER","FLB_HAVE_LUAJIT","FLB_HAVE_C_TLS","FLB_HAVE_ACCEPT4","FLB_HAVE_INOTIFY","FLB_HAVE_GETENTROPY","FLB_HAVE_GETENTROPY_SYS_RANDOM"]}}
So, how someone will notice that there's an error if everything start without problem? in my opinion a daemon should fail if there's anything that may cause an error, just like inside a pod/container, it will be restarted until something it's fixed, this it's more like making an error kind of invisible.

On the other hand, which plugins in Fluent-Bit behaves like what you want to add? Can you elaborate more on why we should implement this? I may be wrong so I want to read more

Thanks in advance!

Hi, I can give some some examples to explain how other out_plugins behave when user give wrong configuration.

example 1(a.config):

[SERVICE]
    Flush        5
    Daemon       False
    Log_Level    error
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020
    Health_Check On

[INPUT]
    name     cpu
    interval_sec 5
    tag      cpu.local

[OUTPUT]
    name http
    host 192.1.1.1
    match  *
    retry_limit 1

[OUTPUT]
    name influxdb
    host 192.2.3.5
    port 8086
    bucket org
    match *
    retry_limit 1

[OUTPUT]
    name opensearch
    host 192.2.2.2
    port 9200
    match *
    retry_limit 1

running config file a.config for several minutes, fluent-bit will not crash, then you will get the following logs:

example 2(b.config):

[SERVICE]
    Flush        5
    Daemon       False
    Log_Level    error
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020
    Health_Check On

[INPUT]
    name     cpu
    interval_sec 5
    tag      cpu.local

[OUTPUT]
    name            pgsql
    host            192.168.142.132
    port            15432
    user            postgres
    password        yourpwd
    match           *
    database        agent
    table           test

[OUTPUT]
    name stdout
    match *

running config file b.config, fluent-bit will exit directly, even if user give correct configuration for the other out_plugins(stdout) and you will get following logs:

example 3(c.config) out_pgsql running with daemon mode:

[SERVICE]
    Flush        5
    Daemon       False
    Log_Level    error
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020
    Health_Check On

[INPUT]
    name     cpu
    interval_sec 5
    tag      cpu.local

[OUTPUT]
    name http
    host 192.1.1.1
    match  *
    retry_limit 1

[OUTPUT]
    name influxdb
    host 192.2.3.5
    port 8086
    bucket org
    match *
    retry_limit 1

[OUTPUT]
    name opensearch
    host 192.2.2.2
    port 9200
    match *
    retry_limit 1

[OUTPUT]
    name            pgsql
    host            192.168.142.132
    port            15432
    user            postgres
    password        yourpwd
    match           *
    database        agent
    table           test
    retry_limit     1
    Daemon          True

running config file c.config for several minutes, fluent-bit will not crash, then you will get the following logs:

TomlinfreeGit · 2023-04-27T02:03:49Z

@TomlinfreeGit Testing this after a couple of minutes, about 10 minutes, I don't see any error in the logs saying that something needs to be fixed, the plugin just finished, doesn't look like a desire behavior since everything will work and will start, but if I can't flush the logs to the desired database this may be an issue.

This it's what I can see in the logs:
Fluent Bit v2.1.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/04/26 12:50:03] [error] [output:pgsql:pgsql.0] failed connecting to host=192.168.2.69 with error: connection to server at "192.168.2.69", port 5432 failed: No route to host
	Is the server running on that host and accepting TCP/IP connections?
And that's it, now on the API I'm getting this:
curl -i http://127.0.0.1:2020/api/v2/health
HTTP/1.1 200 OK
Server: Monkey/1.7.0
Date: Wed, 26 Apr 2023 10:53:51 GMT
Transfer-Encoding: chunked
Content-Type: application/json

{"fluent-bit":{"version":"2.1.0","edition":"Community","flags":["FLB_HAVE_IN_STORAGE_BACKLOG","FLB_HAVE_CHUNK_TRACE","FLB_HAVE_PARSER","FLB_HAVE_RECORD_ACCESSOR","FLB_HAVE_STREAM_PROCESSOR","FLB_HAVE_TLS","FLB_HAVE_OPENSSL","FLB_HAVE_METRICS","FLB_HAVE_WASM","FLB_HAVE_AWS","FLB_HAVE_AWS_CREDENTIAL_PROCESS","FLB_HAVE_SIGNV4","FLB_HAVE_SQLDB","FLB_HAVE_METRICS","FLB_HAVE_HTTP_SERVER","FLB_HAVE_SYSTEMD","FLB_HAVE_VALGRIND","FLB_HAVE_FORK","FLB_HAVE_TIMESPEC_GET","FLB_HAVE_GMTOFF","FLB_HAVE_UNIX_SOCKET","FLB_HAVE_ATTRIBUTE_ALLOC_SIZE","FLB_HAVE_PROXY_GO","FLB_HAVE_LIBBACKTRACE","FLB_HAVE_REGEX","FLB_HAVE_UTF8_ENCODER","FLB_HAVE_LUAJIT","FLB_HAVE_C_TLS","FLB_HAVE_ACCEPT4","FLB_HAVE_INOTIFY","FLB_HAVE_GETENTROPY","FLB_HAVE_GETENTROPY_SYS_RANDOM"]}}
So, how someone will notice that there's an error if everything start without problem? in my opinion a daemon should fail if there's anything that may cause an error, just like inside a pod/container, it will be restarted until something it's fixed, this it's more like making an error kind of invisible.

On the other hand, which plugins in Fluent-Bit behaves like what you want to add? Can you elaborate more on why we should implement this? I may be wrong so I want to read more

Thanks in advance!

Thanks for your kind review.
In your comment, I guess the you didn't turn on the health check function, you can turn on by config the "health_check=on":

[SERVICE]
    Flush        5
    Daemon       False
    Log_Level    error
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020
    Health_Check On

and call http://127.0.0.1:2020/api/v1/health to notice the health status of fluent-bit
https://docs.fluentbit.io/manual/administration/monitoring#health-check-for-fluent-bit

sxd · 2023-05-22T07:44:02Z

@TomlinfreeGit Hi! I was on holidays for the last month so I didn't look into this, I'm getting into this today

TomlinfreeGit · 2023-07-28T02:02:54Z

please do help for a review

@TomlinfreeGit Hi! I was on holidays for the last month so I didn't look into this, I'm getting into this today

TomlinfreeGit · 2023-08-28T08:54:52Z

@sxd Hi please help do a review, if possible, thanks very much!

TomlinfreeGit requested review from sxd, edsiper, leonardo-albertovich, fujimotos and koleini as code owners April 18, 2023 06:09

github-actions bot added the docs-required label Apr 18, 2023

TomlinfreeGit mentioned this pull request Apr 18, 2023

update out_pgsql document fluent/fluent-bit-docs#1081

Open

TomlinfreeGit temporarily deployed to pr April 21, 2023 15:43 — with GitHub Actions Inactive

TomlinfreeGit temporarily deployed to pr April 21, 2023 15:44 — with GitHub Actions Inactive

TomlinfreeGit temporarily deployed to pr April 21, 2023 16:08 — with GitHub Actions Inactive

TomlinfreeGit force-pushed the feature/daemon-out-pgsql branch from b4c4d34 to f3d62a4 Compare April 22, 2023 14:10

TomlinfreeGit changed the title ~~Feature/daemon out pgsql~~ out_pgsql: add a configurable value called "daemon" for out pgsql plugin Apr 22, 2023

edsiper assigned sxd Apr 25, 2023

sxd reviewed Apr 25, 2023

View reviewed changes

plugins/out_pgsql/pgsql.c Outdated Show resolved Hide resolved

sxd reviewed Apr 25, 2023

View reviewed changes

Tom added 2 commits April 26, 2023 09:06

out_pgsql: daemon mode means fb won't exit even pgsql err

82729a0

Signed-off-by: Tom <yao.lin@siemens.com>

out_pgsql: add example config

53ce527

Signed-off-by: Tom <yao.lin@siemens.com>

TomlinfreeGit force-pushed the feature/daemon-out-pgsql branch from f3d62a4 to 53ce527 Compare April 26, 2023 01:07

out_pgsql: code-format &if config err then log err

a221a2f

Signed-off-by: Tom <yao.lin@siemens.com>

TomlinfreeGit requested a review from sxd April 26, 2023 02:18

sxd reviewed Apr 26, 2023

View reviewed changes

plugins/out_pgsql/pgsql.c Show resolved Hide resolved

TomlinfreeGit requested a review from sxd April 28, 2023 01:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

out_pgsql: add a configurable value called "daemon" for out pgsql plugin #7215

out_pgsql: add a configurable value called "daemon" for out pgsql plugin #7215

Uh oh!

TomlinfreeGit commented Apr 18, 2023 •

edited

Loading

Uh oh!

TomlinfreeGit commented Apr 18, 2023 •

edited

Loading

Uh oh!

TomlinfreeGit commented Apr 18, 2023

Uh oh!

edsiper commented Apr 25, 2023

Uh oh!

sxd commented Apr 25, 2023

Uh oh!

Uh oh!

sxd Apr 25, 2023

Uh oh!

TomlinfreeGit Apr 26, 2023

Uh oh!

sxd commented Apr 25, 2023

Uh oh!

TomlinfreeGit commented Apr 26, 2023

Uh oh!

Uh oh!

sxd commented Apr 26, 2023

Uh oh!

TomlinfreeGit commented Apr 27, 2023

Uh oh!

TomlinfreeGit commented Apr 27, 2023

Uh oh!

sxd commented May 22, 2023

Uh oh!

TomlinfreeGit commented Jul 28, 2023

Uh oh!

TomlinfreeGit commented Aug 28, 2023

Uh oh!

Uh oh!

out_pgsql: add a configurable value called "daemon" for out pgsql plugin #7215

Are you sure you want to change the base?

out_pgsql: add a configurable value called "daemon" for out pgsql plugin #7215

Uh oh!

Conversation

TomlinfreeGit commented Apr 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

N/A

Uh oh!

TomlinfreeGit commented Apr 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

example config file for running out_pgsql in daemon mode:

running result and valgrind result:

example config file for running out_pgsql not in daemon mode:

running result and valgrind result:

Uh oh!

TomlinfreeGit commented Apr 18, 2023

Uh oh!

edsiper commented Apr 25, 2023

Uh oh!

sxd commented Apr 25, 2023

Uh oh!

Uh oh!

sxd Apr 25, 2023

Choose a reason for hiding this comment

Uh oh!

TomlinfreeGit Apr 26, 2023

Choose a reason for hiding this comment

Uh oh!

sxd commented Apr 25, 2023

Uh oh!

TomlinfreeGit commented Apr 26, 2023

Uh oh!

Uh oh!

sxd commented Apr 26, 2023

Uh oh!

TomlinfreeGit commented Apr 27, 2023

example 1(a.config):

example 2(b.config):

example 3(c.config) out_pgsql running with daemon mode:

Uh oh!

TomlinfreeGit commented Apr 27, 2023

Uh oh!

sxd commented May 22, 2023

Uh oh!

TomlinfreeGit commented Jul 28, 2023

Uh oh!

TomlinfreeGit commented Aug 28, 2023

Uh oh!

Uh oh!

TomlinfreeGit commented Apr 18, 2023 •

edited

Loading

TomlinfreeGit commented Apr 18, 2023 •

edited

Loading