Can't see fields on Kibana after parsing log.

[spkesan]

Hi,

I was trying to use regexp to parse my logs to extract some of the fields, and I cannot see them on Kibana. I'm using fluentd container to achieve this. But there's no indication from docker logs to know what's wrong.

Here's my fluentd.conf:

<source>
        @type tail
        path /newfluentd/logs/asd.log
        pos_file /newfluentd/logs/asd.log.pos
        tag spk.aeroapp
        read_from_head true
        <parse>
                @type regexp
                expression /^(?<logtime>[a-zA-Z]* \d* \d* \d*:\d*:\d*) GMT\D\d*: INFO \(info\): \(ticker.c:\d*\) {(?<namespace>\w*)} client: tsvc \((?<client_tsvc_error>\d*),(?<client_tsvc_timeout>\d*)\) proxy \((?<client_proxy_complete>\d*),(?<client_proxy_error>\d*),(?<client_proxy_timeout>\d*)\) read \((?<client_read_success>\d*),(?<client_read_error>\d*),(?<client_read_timeout>\d*),(?<client_read_not_found>\d*)\) write \((?<client_write_success>\d*),(?<client_write_error>\d*),(?<client_write_timeout>\d*)\) delete \((?<client_delete_success>\d*),(?<client_delete_error>\d*),(?<client_delete_timeout>\d*),(?<client_delete_not_found>\d*)\) udf \((?<client_udf_complete>\d*),(?<client_udf_error>\d*),(?<client_udf_timeout>\d*)\) lang \((?<client_lang_read_success>\d*),(?<client_lang_write_success>\d*),(?<client_lang_delete_success>\d*),(?<client_lang_error>\d*)\)
                time_key logtime
                time_format %b %d %Y %H:%M:%S
        </parse>
</source>

<match spk.**>
        @type elasticsearch
        host 172.18.0.2
        port 9200
        index_name fluentd_asp
        flush_interval 10s
</match>

Here's my application log line (/newfluentd/logs/asd.log):

Oct 13 2018 23:59:53 GMT+0800: INFO (info): (ticker.c:585) {starhub_kvs} client: tsvc (0,0) proxy (0,0,0) read (1903439312,0,0,2110808) write (345776441,5944686,0) delete (0,0,0,0) udf (0,0,0) lang (0,0,0,0)

Here's my container logs:

2018-11-10 16:41:00 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2018-11-10 16:41:00 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type tail
    path "/newfluentd/logs/asd.log"
    pos_file "/newfluentd/logs/asd.log.pos"
    tag "spk.aeroapp"
    read_from_head true
    <parse>
      @type "regexp"
      expression /^(?<logtime>[a-zA-Z]* \d* \d* \d*:\d*:\d*) GMT\D\d*: INFO \(info\): \(ticker.c:\d*\) {(?<namespace>\w*)} client: tsvc \((?<client_tsvc_error>\d*),(?<client_tsvc_timeout>\d*)\) proxy \((?<client_proxy_complete>\d*),(?<client_proxy_error>\d*),(?<client_proxy_timeout>\d*)\) read \((?<client_read_success>\d*),(?<client_read_error>\d*),(?<client_read_timeout>\d*),(?<client_read_not_found>\d*)\) write \((?<client_write_success>\d*),(?<client_write_error>\d*),(?<client_write_timeout>\d*)\) delete \((?<client_delete_success>\d*),(?<client_delete_error>\d*),(?<client_delete_timeout>\d*),(?<client_delete_not_found>\d*)\) udf \((?<client_udf_complete>\d*),(?<client_udf_error>\d*),(?<client_udf_timeout>\d*)\) lang \((?<client_lang_read_success>\d*),(?<client_lang_write_success>\d*),(?<client_lang_delete_success>\d*),(?<client_lang_error>\d*)\)
      time_key "logtime"
      time_format "%b %d %Y %H:%M:%S"
    </parse>
  </source>
  <match spk.**>
    @type elasticsearch
    host "172.18.0.2"
    port 9200
    index_name "fluentd_asp"
    flush_interval 10s
    <buffer tag>
      flush_mode interval
      retry_type exponential_backoff
      flush_interval 10s
    </buffer>
  </match>
</ROOT>
2018-11-10 16:41:00 +0000 [info]: starting fluentd-1.2.6 pid=7 ruby="2.3.3"
2018-11-10 16:41:00 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby2.3", "-Eascii-8bit:ascii-8bit", "/usr/local/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2018-11-10 16:41:00 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.9.5'
2018-11-10 16:41:00 +0000 [info]: gem 'fluentd' version '1.2.6'
2018-11-10 16:41:00 +0000 [info]: adding match pattern="spk.**" type="elasticsearch"
2018-11-10 16:41:00 +0000 [info]: adding source type="tail"

Can you help in debugging this.

Thanks

[okkez]

Missing slash at the end of the expression parameter.

Please follow the guideline.

[spkesan]

@okkez
Thanks. got that.
Can you also help with the following:

Suppose my logs have multiple lines like this:

Stopped XYZ
Started GET "/users/123/" for 127.0.0.1 at 2013-06-14 12:00:11 +0900
Processing by UsersController#show as HTML
Done Processing ID-254

I need to write regex (in fluent.conf) to parse only the 2nd and 3rd lines and ignore all the other lines:
I did something like this, but it doesn't seem to work. What's the best way.

<parse>
@type regexp
expression /^Started (?<method>[^ ]+) "(?<path>[^"]+)" for (?<host>[^ ]+) at (?<time>[^ ]+ [^ ]+ [^ ]+)/
expression /^Processing by (?<controller>[^\u0023]+)\u0023(?<controller_method>[^ ]+) as (?<format>[^ ]+)/
</parse>

The logs print pattern not match for the other two lines and I cannot see any data on Elasticsearch.

Thanks in advance.

[okkez]

Use the combo with grep Filter Plugin and fluent-plugin-concat

BTW, don't use fluentd repo for asking question.
You can ask your question to the mailing list.