out_stackdriver: Support generic resources. #2992

heartrobotninja · 2021-01-29T07:21:44Z

Added generic_node and generic_task as supported resources.

Of note:
Both credential files or custom metadata server work for auth.
Credential file is preferred, but if you choose the metadata server
then it will need to also return the call for project_id.

Testing covers both of those cases.

Apparently my client was also confused about an old change when I merged my branch which is why METADATA_SERVER_URL gets fixed again here.

Signed-off-by: Joey DeStefanis jdestefanis@google.com

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

Documentation required for this feature

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Added generic_node and generic_task as supported resources. Of note: Both credential files or custom metadata server work for auth. Credential file is preferred, but if you choose the metadata server then it will need to also return the call for project_id. Testing covers both of those cases. Signed-off-by: Joey DeStefanis <jdestefanis@google.com>

heartrobotninja · 2021-01-29T07:38:44Z

Debug output:
$ /usr/local/bin/fluent-bit -c /usr/local/etc/fluent-bit/fluent-bit.conf
Fluent Bit v1.7.0

Copyright (C) 2019-2020 The Fluent Bit Authors
Copyright (C) 2015-2018 Treasure Data
Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
https://fluentbit.io

[2021/01/29 07:34:14] [ info] Configuration:
[2021/01/29 07:34:14] [ info] flush time | 5.000000 seconds
[2021/01/29 07:34:14] [ info] grace | 5 seconds
[2021/01/29 07:34:14] [ info] daemon | 0
[2021/01/29 07:34:14] [ info] ___________
[2021/01/29 07:34:14] [ info] inputs:
[2021/01/29 07:34:14] [ info] cpu
[2021/01/29 07:34:14] [ info] ___________
[2021/01/29 07:34:14] [ info] filters:
[2021/01/29 07:34:14] [ info] ___________
[2021/01/29 07:34:14] [ info] outputs:
[2021/01/29 07:34:14] [ info] stackdriver.0
[2021/01/29 07:34:14] [ info] ___________
[2021/01/29 07:34:14] [ info] collectors:
[2021/01/29 07:34:14] [ info] [engine] started (pid=1053463)
[2021/01/29 07:34:14] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2021/01/29 07:34:14] [debug] [storage] [cio stream] new stream registered: cpu.0
[2021/01/29 07:34:14] [ info] [storage] version=1.1.0, initializing...
[2021/01/29 07:34:14] [ info] [storage] in-memory
[2021/01/29 07:34:14] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/01/29 07:34:14] [debug] [stackdriver:stackdriver.0] created event channels: read=18 write=19
[2021/01/29 07:34:14] [ info] [output:stackdriver:stackdriver.0] metadata_server set to http://metadata.google.internal
[2021/01/29 07:34:14] [ warn] [output:stackdriver:stackdriver.0] client_email is not defined, using a default one
[2021/01/29 07:34:14] [ warn] [output:stackdriver:stackdriver.0] private_key is not defined, fetching it from metadata server
[2021/01/29 07:34:14] [debug] [http_client] not using http_proxy for header
[2021/01/29 07:34:14] [debug] [http_client] header=GET /computeMetadata/v1/instance/service-accounts/default/token HTTP/1.1
Host: metadata.google.internal:80
Content-Length: 0
User-Agent: Fluent-Bit
Content-Type: application/text
Metadata-Flavor: Google

[2021/01/29 07:34:14] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2021/01/29 07:34:14] [debug] [upstream] KA connection #20 to metadata.google.internal:80 is now available
[2021/01/29 07:34:14] [debug] [upstream] KA connection #20 to metadata.google.internal:80 has been assigned (recycled)
[2021/01/29 07:34:14] [debug] [http_client] not using http_proxy for header
[2021/01/29 07:34:14] [debug] [http_client] header=GET /computeMetadata/v1/project/project-id HTTP/1.1
Host: metadata.google.internal:80
Content-Length: 0
User-Agent: Fluent-Bit
Content-Type: application/text
Metadata-Flavor: Google

[2021/01/29 07:34:14] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2021/01/29 07:34:14] [debug] [upstream] KA connection #20 to metadata.google.internal:80 is now available
[2021/01/29 07:34:14] [debug] [router] match rule cpu.0:stackdriver.0
[2021/01/29 07:34:14] [ info] [sp] stream processor started
[2021/01/29 07:34:18] [debug] [task] created task=0x7f35801d4520 id=0 OK
[2021/01/29 07:34:18] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload
[2021/01/29 07:34:18] [debug] [http_client] not using http_proxy for header
[2021/01/29 07:34:18] [debug] [http_client] header=POST /v2/entries:write HTTP/1.1
Host: logging.googleapis.com
Content-Length: 1287
User-Agent: Fluent-Bit
Content-Type: application/json
Authorization: ...

[2021/01/29 07:34:29] [debug] [output:stackdriver:stackdriver.0] HTTP Status=200
[2021/01/29 07:34:29] [debug] [out coro] cb_destroy coro_id=3
[2021/01/29 07:34:29] [debug] [task] destroy task=0x7f35802c44d0 (task_id=0)
[2021/01/29 07:34:34] [ info] [engine] service stopped

heartrobotninja · 2021-01-29T07:40:57Z

Example Config (relevant part)

w/ metadata server (which works fine on GCE with the default GCE metadata server)
[OUTPUT]
name stackdriver
match *
resource generic_node
location test
node_id test-node
namespace test-namespace

w/ cred file
[OUTPUT]
name stackdriver
match *
google_service_credentials file.json
resource generic_node
location test
node_id test-node
namespace test-namespace

Checking resource caused a memory leak, but checking node_id for non-null fixes that for both generic_node and generic_task. Signed-off-by: Joey DeStefanis <jdestefanis@google.com>

heartrobotninja · 2021-01-29T08:06:07Z

Valgrind output:

valgrind --leak-check=full /usr/local/bin/fluent-bit -c /usr/local/etc/fluent-bit/fluent-bit.conf
==1056784== Memcheck, a memory error detector
==1056784== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==1056784== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==1056784== Command: /usr/local/bin/fluent-bit -c /usr/local/etc/fluent-bit/fluent-bit.conf
==1056784==
Fluent Bit v1.7.0

Copyright (C) 2019-2020 The Fluent Bit Authors
Copyright (C) 2015-2018 Treasure Data
Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
https://fluentbit.io

[2021/01/29 08:02:58] [ info] [engine] started (pid=1056784)
[2021/01/29 08:02:58] [ info] [storage] version=1.1.0, initializing...
[2021/01/29 08:02:58] [ info] [storage] in-memory
[2021/01/29 08:02:58] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/01/29 08:02:58] [ info] [output:stackdriver:stackdriver.0] metadata_server set to http://metadata.google.internal
[2021/01/29 08:02:58] [ warn] [output:stackdriver:stackdriver.0] client_email is not defined, using a default one
[2021/01/29 08:02:58] [ warn] [output:stackdriver:stackdriver.0] private_key is not defined, fetching it from metadata server
[2021/01/29 08:02:59] [ info] [sp] stream processor started
^C[2021/01/29 08:03:05] [engine] caught signal (SIGINT)
[2021/01/29 08:03:05] [ info] [input] pausing cpu.0
[2021/01/29 08:03:05] [ warn] [engine] service will stop in 5 seconds
[2021/01/29 08:03:10] [ info] [engine] service stopped
==1056784==
==1056784== HEAP SUMMARY:
==1056784== in use at exit: 0 bytes in 0 blocks
==1056784== total heap usage: 72,276 allocs, 72,276 frees, 12,899,100 bytes allocated
==1056784==
==1056784== All heap blocks were freed -- no leaks are possible
==1056784==
==1056784== For lists of detected and suppressed errors, rerun with: -s
==1056784== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

heartrobotninja · 2021-01-29T08:06:47Z

Documentation update is waiting for fluent/fluent-bit-docs#444 to be merged first.

heartrobotninja · 2021-01-29T08:07:20Z

@igorpeshansky or @hsmatulisgoogle Could you have a look here?

hsmatulis

The logic looks fine, although I have a couple nitpicks

plugins/out_stackdriver/stackdriver_conf.c

plugins/out_stackdriver/stackdriver.h

Also remove a wild newline. Signed-off-by: Joey DeStefanis <jdestefanis@google.com>

igorpeshansky

Some minor comments.

plugins/out_stackdriver/stackdriver.c

plugins/out_stackdriver/stackdriver.h

plugins/out_stackdriver/stackdriver_conf.c

tests/runtime/out_stackdriver.c

plugins/out_stackdriver/stackdriver_conf.c

VSCode strikes again. Signed-off-by: Joey DeStefanis <jdestefanis@google.com>

igorpeshansky

LGTM

edsiper · 2021-02-02T04:39:17Z

thanks everyone

heartrobotninja · 2021-02-11T23:26:13Z

Doc update: fluent/fluent-bit-docs#459

heartrobotninja requested review from edsiper, fujimotos and koleini as code owners January 29, 2021 07:21

out_stackdriver: Fix memory leak

94eb90c

Checking resource caused a memory leak, but checking node_id for non-null fixes that for both generic_node and generic_task. Signed-off-by: Joey DeStefanis <jdestefanis@google.com>

hsmatulis reviewed Jan 29, 2021

View reviewed changes

plugins/out_stackdriver/stackdriver_conf.c Outdated Show resolved Hide resolved

plugins/out_stackdriver/stackdriver.h Outdated Show resolved Hide resolved

out_stackdriver: Add is_ prefix to resource bools.

1d5307b

Also remove a wild newline. Signed-off-by: Joey DeStefanis <jdestefanis@google.com>

hsmatulis approved these changes Jan 29, 2021

View reviewed changes

igorpeshansky suggested changes Jan 29, 2021

View reviewed changes

out_stackdriver: Fix whitespace issues.

a72cd0d

VSCode strikes again. Signed-off-by: Joey DeStefanis <jdestefanis@google.com>

igorpeshansky approved these changes Feb 1, 2021

View reviewed changes

edsiper merged commit fb1312c into fluent:master Feb 2, 2021

out_stackdriver: Support generic resources. #2992

out_stackdriver: Support generic resources. #2992

Uh oh!

Conversation

heartrobotninja commented Jan 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

heartrobotninja commented Jan 29, 2021

Uh oh!

heartrobotninja commented Jan 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

heartrobotninja commented Jan 29, 2021

Uh oh!

heartrobotninja commented Jan 29, 2021

Uh oh!

heartrobotninja commented Jan 29, 2021

Uh oh!

hsmatulis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

igorpeshansky left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

igorpeshansky left a comment

Choose a reason for hiding this comment

Uh oh!

edsiper commented Feb 2, 2021

Uh oh!

heartrobotninja commented Feb 11, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

heartrobotninja commented Jan 29, 2021 •

edited

Loading

heartrobotninja commented Jan 29, 2021 •

edited

Loading