out_stackdriver: Make metadata url configurable #2944

heartrobotninja · 2021-01-14T02:00:43Z

Allows users to configure the metadata server url via config file
and environment variable. If neither are supplied it will fall-back to
http://metadata.google.internal.

Evaluation order:

Config File via metadata_server_url
Environment Variable via METADATA_SERVER_URL
Internally set fallback variable FLB_STD_METADATA_SERVER_URL

Evaluating the config file before the environment variable is a
design decision in relation to security as it is easier for a bad
actor to set a environment variable than manipulate the config file.

Signed-off-by: Joey DeStefanis jdestefanis@google.com

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

Documentation

Documentation required for this feature

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

heartrobotninja · 2021-01-14T04:29:39Z

@igorpeshansky

heartrobotninja · 2021-01-14T04:36:20Z

Example Config File:

[SERVICE]
Flush 1
Daemon Off
Log_Level info

[INPUT]
Name cpu
Tag cpu.local

[OUTPUT]
Name stackdriver
Match *
Metadata_Server_URL http://metadata-in-config.whatever

heartrobotninja · 2021-01-14T04:39:16Z

Debug Output (expected to fail as I am not running a metadata server locally}:

fluent-bit -c ~/fluent-bit/fluent-bit.conf
Fluent Bit v1.7.0

Copyright (C) 2019-2020 The Fluent Bit Authors
Copyright (C) 2015-2018 Treasure Data
Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
https://fluentbit.io

[2021/01/13 20:38:01] [ info] Configuration:
[2021/01/13 20:38:01] [ info] flush time | 1.000000 seconds
[2021/01/13 20:38:01] [ info] grace | 5 seconds
[2021/01/13 20:38:01] [ info] daemon | 0
[2021/01/13 20:38:01] [ info] ___________
[2021/01/13 20:38:01] [ info] inputs:
[2021/01/13 20:38:01] [ info] cpu
[2021/01/13 20:38:01] [ info] ___________
[2021/01/13 20:38:01] [ info] filters:
[2021/01/13 20:38:01] [ info] ___________
[2021/01/13 20:38:01] [ info] outputs:
[2021/01/13 20:38:01] [ info] stackdriver.0
[2021/01/13 20:38:01] [ info] ___________
[2021/01/13 20:38:01] [ info] collectors:
[2021/01/13 20:38:01] [ info] [engine] started (pid=2489620)
[2021/01/13 20:38:01] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2021/01/13 20:38:01] [debug] [storage] [cio stream] new stream registered: cpu.0
[2021/01/13 20:38:01] [ info] [storage] version=1.1.0, initializing...
[2021/01/13 20:38:01] [ info] [storage] in-memory
[2021/01/13 20:38:01] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/01/13 20:38:01] [debug] [stackdriver:stackdriver.0] created event channels: read=18 write=19
[2021/01/13 20:38:01] [ info] [output:stackdriver:stackdriver.0] metadata_url set to http://metadata-in-config.whatever
[2021/01/13 20:38:01] [ warn] [output:stackdriver:stackdriver.0] client_email is not defined, using a default one
[2021/01/13 20:38:01] [ warn] [output:stackdriver:stackdriver.0] private_key is not defined, fetching it from metadata server
[2021/01/13 20:38:01] [ warn] [net] getaddrinfo(host='metadata-in-config.whatever'): Name or service not known
[2021/01/13 20:38:01] [debug] [upstream] connection #-1 failed to metadata-in-config.whatever:80
[2021/01/13 20:38:01] [error] [output:stackdriver:stackdriver.0] failed to create metadata connection
[2021/01/13 20:38:01] [error] [output:stackdriver:stackdriver.0] can't fetch token from the metadata server
[2021/01/13 20:38:01] [ warn] [output:stackdriver:stackdriver.0] token retrieval failed
[2021/01/13 20:38:01] [ warn] [net] getaddrinfo(host='metadata-in-config.whatever'): Name or service not known
[2021/01/13 20:38:01] [debug] [upstream] connection #-1 failed to metadata-in-config.whatever:80
[2021/01/13 20:38:01] [error] [lib] backend failed
[2021/01/13 20:38:01] [error] [output:stackdriver:stackdriver.0] failed to create metadata connection
[2021/01/13 20:38:01] [error] [output:stackdriver:stackdriver.0] can't fetch project id from the metadata server
[2021/01/13 20:38:01] [error] [output] Failed to initialize 'stackdriver' plugin
[2021/01/13 20:38:01] [ info] [input] pausing cpu.0

heartrobotninja · 2021-01-14T05:17:12Z

Documentation: fluent/fluent-bit-docs#444

heartrobotninja · 2021-01-14T05:35:26Z

Looking at the CI/Appveyor failures, I don't see anything that would indicate the errors are from my changes specifically, can you all have a look?

heartrobotninja · 2021-01-20T22:34:17Z

Compiled and ran on a GCE instance, no valgrind errors.

$ valgrind /usr/local/bin/fluent-bit -c /usr/local/etc/fluent-bit/fluent-bit.conf
==234435== Memcheck, a memory error detector
==234435== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==234435== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==234435== Command: /usr/local/bin/fluent-bit -c /usr/local/etc/fluent-bit/fluent-bit.conf
==234435==
Fluent Bit v1.7.0

Copyright (C) 2019-2020 The Fluent Bit Authors
Copyright (C) 2015-2018 Treasure Data
Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
https://fluentbit.io

[2021/01/20 22:33:16] [ info] [engine] started (pid=234435)
[2021/01/20 22:33:16] [ info] [storage] version=1.1.0, initializing...
[2021/01/20 22:33:16] [ info] [storage] in-memory
[2021/01/20 22:33:16] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/01/20 22:33:16] [ info] [output:stackdriver:stackdriver.0] metadata_url set to http://metadata.google.internal
[2021/01/20 22:33:16] [ warn] [output:stackdriver:stackdriver.0] client_email is not defined, using a default one
[2021/01/20 22:33:16] [ warn] [output:stackdriver:stackdriver.0] private_key is not defined, fetching it from metadata server
[2021/01/20 22:33:17] [ info] [sp] stream processor started
^C[2021/01/20 22:33:27] [engine] caught signal (SIGINT)
[2021/01/20 22:33:27] [ info] [input] pausing cpu.0
[2021/01/20 22:33:27] [ warn] [engine] service will stop in 5 seconds
[2021/01/20 22:33:32] [ info] [engine] service stopped
==234435==
==234435== HEAP SUMMARY:
==234435== in use at exit: 0 bytes in 0 blocks
==234435== total heap usage: 36,993 allocs, 36,993 frees, 12,084,184 bytes allocated
==234435==
==234435== All heap blocks were freed -- no leaks are possible
==234435==
==234435== For lists of detected and suppressed errors, rerun with: -s
==234435== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

heartrobotninja · 2021-01-25T19:27:25Z

@edsiper Can I get this PR looked at? It is ready.

edsiper · 2021-01-27T21:49:04Z

@hsmatulisgoogle @igorpeshansky

can you take a look at this PR from a logic/usability perspective ?

igorpeshansky

One naming comment, and a few formatting ones.

plugins/out_stackdriver/stackdriver.h

plugins/out_stackdriver/stackdriver_conf.c

tests/runtime/out_stackdriver.c

heartrobotninja · 2021-01-28T06:10:40Z

Done fixing initial round of comments and updated documentation.

edsiper · 2021-01-28T15:24:26Z

if @hsmatulisgoogle @igorpeshansky approve this we can merge it

plugins/out_stackdriver/fluent-bit.code-workspace

heartrobotninja · 2021-01-28T20:15:26Z

I am not sure why its saying I have incorrectly signed off as I ran git commit -sa like every time before.

heartrobotninja · 2021-01-28T20:16:56Z

I am going to roll that back, somehow the merge I did brought a bunch of unnecessary things in.

heartrobotninja · 2021-01-28T20:53:44Z

I had to revert and then merging back in was... a problem. Fixed that with a force push, but DCO is still mad at my vscode deletion push, even though it has signoff.

Allows users to configure the metadata server url via config file and environment variable. If neither are supplied it will fall-back to http://metadata.google.internal. Evaluation order: 1. Config File 2. Environment Variable 3. Internally set fallback variable Evaluating the config file before the environment variable is a design decision in relation to security as it is easier for a bad actor to set a environment variable than manipulate the config file. Signed-off-by: Joey DeStefanis <jdestefanis@google.com> [out_stackdriver] Fix blank lines and naming. Signed-off-by: Joey DeStefanis <jdestefanis@google.com> [out_stackdriver] Delete vscode file. Signed-off-by: Joey DeStefanis <jdestefanis@google.com> Revert "[out_stackdriver] Delete vscode file." This reverts commit 7a6dcd2. out_stackdriver: Make metadata url configurable Allows users to configure the metadata server url via config file and environment variable. If neither are supplied it will fall-back to http://metadata.google.internal. Evaluation order: 1. Config File 2. Environment Variable 3. Internally set fallback variable Evaluating the config file before the environment variable is a design decision in relation to security as it is easier for a bad actor to set a environment variable than manipulate the config file. Signed-off-by: Joey DeStefanis <jdestefanis@google.com> [out_stackdriver] Fix blank lines and naming. Signed-off-by: Joey DeStefanis <jdestefanis@google.com> out_stackdriver: Make metadata url configurable Allows users to configure the metadata server url via config file and environment variable. If neither are supplied it will fall-back to http://metadata.google.internal. Evaluation order: 1. Config File 2. Environment Variable 3. Internally set fallback variable Evaluating the config file before the environment variable is a design decision in relation to security as it is easier for a bad actor to set a environment variable than manipulate the config file. Signed-off-by: Joey DeStefanis <jdestefanis@google.com> [out_stackdriver] Fix blank lines and naming. Signed-off-by: Joey DeStefanis <jdestefanis@google.com> [out_stackdriver] Accidentally un-ignored vscode file. Signed-off-by: Joey DeStefanis <jdestefanis@google.com> [out_stackdriver] remove conflict markers. It appears that conflict markers weren't removed fully before the save and push. Signed-off-by: Joey DeStefanis <jdestefanis@google.com>

edsiper · 2021-01-28T22:41:37Z

thanks everyone!

heartrobotninja requested review from edsiper, fujimotos and koleini as code owners January 14, 2021 02:00

heartrobotninja mentioned this pull request Jan 14, 2021

out_stackdriver: Document metadata_server_url config option fluent/fluent-bit-docs#444

Merged

This comment has been minimized.

Sign in to view

igorpeshansky suggested changes Jan 28, 2021

View reviewed changes

plugins/out_stackdriver/stackdriver.h Outdated Show resolved Hide resolved

plugins/out_stackdriver/stackdriver_conf.c Outdated Show resolved Hide resolved

tests/runtime/out_stackdriver.c Show resolved Hide resolved

hsmatulis reviewed Jan 28, 2021

View reviewed changes

plugins/out_stackdriver/fluent-bit.code-workspace Outdated Show resolved Hide resolved

heartrobotninja requested a review from PettitWesley as a code owner January 28, 2021 20:11

heartrobotninja force-pushed the master branch 2 times, most recently from 78468f8 to 84d643a Compare January 28, 2021 20:51

heartrobotninja force-pushed the master branch from 3e7e586 to b8954a8 Compare January 28, 2021 22:24

hsmatulis approved these changes Jan 28, 2021

View reviewed changes

edsiper merged commit 0650bec into fluent:master Jan 28, 2021

out_stackdriver: Make metadata url configurable #2944

out_stackdriver: Make metadata url configurable #2944

Uh oh!

Conversation

heartrobotninja commented Jan 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

heartrobotninja commented Jan 14, 2021

Uh oh!

heartrobotninja commented Jan 14, 2021

Uh oh!

heartrobotninja commented Jan 14, 2021

Uh oh!

heartrobotninja commented Jan 14, 2021

Uh oh!

This comment has been minimized.

heartrobotninja commented Jan 14, 2021

Uh oh!

heartrobotninja commented Jan 20, 2021

Uh oh!

heartrobotninja commented Jan 25, 2021

Uh oh!

edsiper commented Jan 27, 2021

Uh oh!

igorpeshansky left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

heartrobotninja commented Jan 28, 2021

Uh oh!

edsiper commented Jan 28, 2021

Uh oh!

Uh oh!

heartrobotninja commented Jan 28, 2021

Uh oh!

heartrobotninja commented Jan 28, 2021

Uh oh!

heartrobotninja commented Jan 28, 2021

Uh oh!

edsiper commented Jan 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

heartrobotninja commented Jan 14, 2021 •

edited

Loading