Can’t parse haproxy logs

[manuelsaks]

I created Grok pattern which works in Kibana Debug Grok devtool:
Sample data:

[May 22 02:22:22 server1 haproxy[5089]: -:22222 [22/May/2021:02:22:22.222] www-https~ myapp/node2 site.domain.com 0/0/0/18/18 200 200 - - ---- 222/222/2/0/0 0/0 \"OPTIONS /api/v2/app/ HTTP/1.1\"]

Grok pattern:

%{HAPROXY_LOG_DATE:haproxy.logdate} %{NOTSPACE:haproxy.host} %{NOTSPACE:process.name[pid]}[%{NUMBER:process.pid:long}] (%{IP:source.address}|-):%{POSINT:source} %{HAPROXY_DATE:haproxy.request_date} %{NOTSPACE:haproxy.frontend_name} %{NOTSPACE:haproxy.backend_name}/%{NOTSPACE:haproxy.server_name} %{NOTSPACE:haproxy.http.captured.request.headers} %{NUMBER:haproxy.http.request.time_wait_ms:long}/%{NUMBER:haproxy.total_waiting_time_ms:long}/%{NUMBER:haproxy.connection_wait_time_ms:long}/%{NUMBER:haproxy.http.request.time_wait_without_data_ms:long}/%{NUMBER:temp.duration:long} %{NUMBER:http.response.status_code:long} %{NUMBER:haproxy.bytes_read:long} %{NOTSPACE:haproxy.http.request.captured_cookie} %{NOTSPACE:haproxy.http.response.captured_cookie} %{NOTSPACE:haproxy.termination_state} %{NUMBER:haproxy.connections.active:long}/%{NUMBER:haproxy.connections.frontend:long}/%{NUMBER:haproxy.connections.backend:long}/%{NUMBER:haproxy.connections.server:long}/%{NUMBER:haproxy.connections.retries:long} %{NUMBER:haproxy.server_queue:long}/%{NUMBER:haproxy.backend_queue:long} \\\"%{NOTSPACE:haproxy.http.request.method} %{NOTSPACE:haproxy.http.request.captured_headers} %{NOTSPACE:haproxy.http.response.captured_headers}\\\"

Custom patterns:

HAPROXY_LOG_DATE %{MONTH} %{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}
HAPROXY_DATE \[%{MONTHDAY}[/-]%{MONTH}[/-]%{YEAR}:%{HOUR}:%{MINUTE}:%{SECOND}\]

Structured data:

{
  "process": {
    "name[pid]": "haproxy[5089]"
  },
  "temp": {
    "duration": 18
  },
  "haproxy": {
    "server_name": "node2",
    "total_waiting_time_ms": 0,
    "termination_state": "----",
    "connection_wait_time_ms": 0,
    "bytes_read": 200,
    "backend_queue": 0,
    "backend_name": "myapp",
    "logdate": "May 22 02:22:22",
    "host": "server1",
    "request_date": "[22/May/2021:02:22:22.222]",
    "http": {
      "request": {
        "captured_cookie": "-",
        "time_wait_without_data_ms": 18,
        "captured_headers": "/api/v2/app/",
        "method": "OPTIONS",
        "time_wait_ms": 0
      },
      "response": {
        "captured_cookie": "-",
        "captured_headers": "HTTP/1.1"
      },
      "captured": {
        "request": {
          "headers": "site.domain.com"
        }
      }
    },
    "frontend_name": "www-https~",
    "server_queue": 0,
    "connections": {
      "server": 0,
      "retries": 0,
      "active": 222,
      "backend": 2,
      "frontend": 222
    }
  },
  "http": {
    "response": {
      "status_code": 200
    }
  },
  "source": "22222"
}

First I thought that something else causes the problem: discuss.elastic.co

But the pattern is working, and I don't know why. but it doesn't work with Filebeat and Elastic.
I have an error in Elastic:

Provided Grok expressions do not match field value:
[May 22 02:22:22 server1 haproxy[5089]: -:22222 [22/May/2021:02:22:22.222] www-https~ myapp/node2 site.domain.com 0/0/0/18/18 200 200 - - ---- 222/222/2/0/0 0/0 \"OPTIONS /api/v2/app/ HTTP/1.1\"]

That's my current config file: gist.github.com

• Version: Filebeat 7.6.1; Elastic&Kibana 7.7.1
• Operating System: Debian Buster
• Discuss Forum URL: discuss.elastic.co

[manuelsaks]

I have the same error. I tried also to upgrade filebeat to 7.12.1 version and copy your pipline.json.
If you have some questions about my issue feel free to ask.

[legoguy1000]

how did you test the changes? I would install the latest filebeat and then update that pipeline with the changes above. I added the sample log that you provided in the ticket. If there are other logs that don't work let me know.

[legoguy1000]

I have the same error. I tried also to upgrade filebeat to 7.12.1 version and copy your pipline.json.
If you have some questions about my issue feel free to ask.

@maarsaks can you provide more details on what didn't work. I added ur sample log to the tests and it parsed properly.

[manuelsaks]

how did you test the changes? I would install the latest filebeat and then update that pipeline with the changes above. I added the sample log that you provided in the ticket. If there are other logs that don't work let me know.

I've tested on docker images with docker-compose using the latest, fresh version of filebeat.

I have the same error. I tried also to upgrade filebeat to 7.12.1 version and copy your pipline.json.
If you have some questions about my issue feel free to ask.

@maarsaks can you provide more details on what didn't work. I added ur sample log to the tests and it parsed properly.

I'm using Kibana to filter logs and there's is an error: 'Provided Grok expressions do not match field value'.
What details exactly do you need?

[legoguy1000]

How did you update the pipeline? Is the log thats having the issue the same that I added to the sample data??

[manuelsaks]

How did you update the pipeline?

I updated pipeline by replacing default config with yours in a new, freshly installed filebeat under /usr/share/filebeat/module/haproxy/log/pipline.json.

Is the log thats having the issue the same that I added to the sample data??

Yes.

[legoguy1000]

I updated pipeline by replacing default config with yours in a new, freshly installed filebeat under /usr/share/filebeat/module/haproxy/log/pipline.json.
Ok, and the pipeline loaded into Elasticsearch matches? I've never modified a pipeline that way. I would have done it through Kibana dev tools/API after it was loaded. Can you post the resulting event from ES?

[manuelsaks]

I updated pipeline by replacing default config with yours in a new, freshly installed filebeat under /usr/share/filebeat/module/haproxy/log/pipline.json.
Ok, and the pipeline loaded into Elasticsearch matches? I've never modified a pipeline that way. I would have done it through Kibana dev tools/API after it was loaded. Can you post the resulting event from ES?

Yup, it loaded and this is the message from ES:

Provided Grok expressions do not match field value:
[May 22 02:22:22 server1 haproxy[5089]: -:22222 [22/May/2021:02:22:22.222] www-https~ myapp/node2 site.domain.com 0/0/0/18/18 200 200 - - ---- 222/222/2/0/0 0/0 \"OPTIONS /api/v2/app/ HTTP/1.1\"]

[legoguy1000]

Hmm, IDK. Here, https://github.com/elastic/beats/pull/25835/files#diff-8363329fdb5b538e99f1112198aaa68d0a353c161a49f26a1041f6f49ae47cfdR63, you can see that the sample log is the same as u provided and is being parsed properly.

[manuelsaks]

Hmm, IDK. Here, https://github.com/elastic/beats/pull/25835/files#diff-8363329fdb5b538e99f1112198aaa68d0a353c161a49f26a1041f6f49ae47cfdR63, you can see that the sample log is the same as u provided and is being parsed properly.

And that's the problem. Like I said in my first comment the pattern is working in Grok debugger, but it doesn't display correctly in Kibana.

[manuelsaks]

@legoguy1000

I've created the repo how to reproduce the error in ~5 minutes - make sure that you've made the all steps from README.md.
github.com/maarsaks/elk-docker-compose