stdin should support non JSON string

[mp3monster]

Is your feature request related to a problem? Please describe.
12 Factor app says we should push our log messages to stdout, which can be captured by Fluentd for example. However, using Fluentbit to capture stdout on stdin, as illustrated, only works if the incoming text is in a simple JSON format. This is very restrictive and constrains the stdin use case to a very small number of use cases.

Describe the solution you'd like
There are a couple of possible options:

1. provide the option that allows the entire input to be captured as the log event using a defined delimiter character or characters (e.g. newline, linefeed, or any literal value
2. for non JSON payloads, use a parser to determine what each record is
3. option for a Lua parser to be incorporated to break the stdin stream into records
4. Define the number of characters each record gets - while not ideal - it would make it possible to still capture a stdin stream of non JSON

I would suggest option 1 is the simplest option which wont compromise performance and would be relatively easy to introduce

Describe alternatives you've considered
See above

Additional context
A simple test scenario would be a variation of the stdin example so that the script-generating events might be:

#!/bin/sh

while :; do
  echo -n "blah blah blah\n"
  sleep 1
done

The configuration could then be:

[input]
  name stdin
  terminator \n
  tag capture_non_json

we could then deploy and run along the lines of:

./script.sh | fluent-bit -c myConfig.conf

This would allow us to support the principles of the 12-factor app more elegantly. It would avoid having to pipe stdout to a log file and the issues of piping output would have with log rotation (or absence of)

[leonardo-albertovich]

It seems that we need to improve the documentation and a PR would be highly appreciated.

While in_stdin does not support plaintext terminator based ingestion at the moment in a way that's comparably as simple as in_tcp it does support parsers through the parser setting which operates in the exact same way it does in other plugins.

If you really need to use a separator based parser for raw lines the best option I can think of (albeit slightly wasteful) would be defining a regular expression parser like this :

parser.conf

[PARSER]
    Name   plaintext
    Format regex
    Regex  (?<msg>([^\n*])+)

fluent-bit.conf

[SERVICE]
    flush 1
    grace 1
    log_level info
    http_server off
    parsers_file parser.conf

[INPUT]
    name stdin
    parser plaintext

[OUTPUT]
    name stdout
    match *

I hope this helps and please, if you can, send that documentation PR.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.