Environment Variables are not evaluated before @INCLUDE

[ahollist]

Bug Report

Describe the bug
When setting the main configuration file to use a different output configuration file based on environment variables, the @ INCLUDE directive does not parse the value of environment variables and instead just directly uses the written line in the configuration file.

To Reproduce
$ export LOG_LEVEL=DEBUG
main.conf:

[SERVICE]
...
[INPUT]
...

@INCLUDE output_match_${LOG_LEVEL}.conf

output_match_DEBUG.conf:

[OUTPUT]
    Name    stdout
    Match   *

Terminal output:
$ /opt/td-agent-bit/bin/td-agent-bit -c /etc/td-agent-bit/main.conf

Fluent Bit v1.3.10
Copyright (C) Treasure Data

[2020/03/10 01:45:10] [Warning] [config] I cannot open /etc/td-agent-bit//output_match_${LOG_LEVEL}.conf file
Error: Configuration file contains errors. Aborting

Expected behavior
The environment variable should expand to its value and load output_match_DEBUG.conf

Your Environment

• Version used: v1.3.10 td-agent-bit
• Environment name and version: td-agent-bit via fluent bit apt servers
• Server type and version: Raspberry Pi3
• Operating System and version: Raspbian Debian Buster
• Filters and plugins: no filters or outputs are relevant here

Additional context

I have several different matches and outputs I would like to use when various environment variables are set, and I am unable to figure out a way how, though if there is a preferred different way of doing this, I am happy to learn how.

I noticed that there is an "inline config" cli input option for Fluentd, but as far as I can tell there is no equivalent for fluent bit.

Note I am calling this directly using the td-agent-bit binary, and I experienced the same issue when attempting to launch through a service unit file using the EnvironmentFile option as well.

[acarsercan]

Maybe I misunderstood this, but are you suggesting your ENV are not picked up by td-agent-bit? If so, I'm having the exact same problem with 1.4.2

[ahollist]

Environment Variables used in filter/parser/input/output sections are used correctly, but if I try to use a variable in an @include directive it is not evaluated to its value.

[edsiper]

thanks, we will fix that shortly

[adeelamin15]

@edsiper Can you assign this to me?

[koalalam]

I got similar issue in version 1.4.2 - is environment variable supported in Parser File?

[marcosflobo]

Good catch on this @ahollist .

Is there any vision about which version this enhancement will come?

[NachoRaik]

Hi @edsiper , was there any news around this issue?

[frosiere]

This issue would definitly help handling conditional outputs.
Can we expect it in the next release of Fluent-bit?
Any workaround?
Thanks for your help.

[garciarodrigor]

Hi @edsiper , was there any news around this issue? We have the same issue, any workaround?
Thanks for your help.

[Clausewitz45]

Hi, I created a feature request to be able to use Environment variables in Tag also - maybe if this will be fixed, 5240 can be addressed also.

[stevenolen]

I'm also going to have to do some awkward file management since this is missing. any update?