Description
Please post all questions and issues on https://discuss.elastic.co/c/beats
before opening a Github Issue. Your questions will reach a wider audience there,
and if we confirm that there is a bug, then you can open a new issue.
For security vulnerabilities please only send reports to security@elastic.co.
See https://www.elastic.co/community/security for more information.
For confirmed bugs, please report:
- Version: 6.1.1
- Operating System:
Linux 4.4.0-31-generic [question] Failed to publish topology #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux - Steps to Reproduce:
Traefik access log example:
94.254.131.115 - - [19/Jan/2018:10:01:02 +0000] "GET /assets/52f8f2e711d235d76044799e/owners?oauth_token=ya29.GltABOXd_gtG-XVvYX2YhxXJiXVvbHRMXn9fbzc_mDfl2rDhqK0CrAlwuwwRWnNnEaMDwkmyI7-QGbRSB0Hzje2cc__FjTQ1iuiYTSIBaIPfxSWip5jx6zqvsVVo HTTP/1.1" 200 85 - "Android" 623112 "Host-api-wearerealitygames-com-2" "http://172.25.0.9:4140" 13ms
89.64.35.193 - - [19/Jan/2018:10:01:02 +0000] "GET /marketplace/tax?oauth_token=ya29.Gl0fBWnrJ7DcEU-tN-O3Vxn2XZVaz2I-hFTjP1JQzhYFVT-SKtlmo9hSzrx3n82LUwUxJ1s5lmU8U3Mc9gA_aCxBk49ShYEwvmYOWxJJyldDIJ7hY4us4LoiSY1OqAM HTTP/1.1" 200 150 - "Android" 623114 "Host-api-wearerealitygames-com-2" "http://172.25.0.6:4140" 8ms
Grok pattern:
link: https://github.com/elastic/beats/blob/master/filebeat/module/traefik/access/ingest/pipeline.json#L7
pattern:
%{IPORHOST:traefik.access.remote_ip} - %{DATA:traefik.access.user_name} \[%{HTTPDATE:traefik.access.time}\] "%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP%{NUMBER:traefik.access.http_version}" %{NUMBER:traefik.access.response_code} (?:%{NUMBER:traefik.access.body_sent.bytes}|-)( "%{DATA:traefik.access.referrer}")?( "%{DATA:traefik.access.agent}")?(?:%{NUMBER:traefik.access.request_count}|-)?( "%{DATA:traefik.access.frontend_name}")?( "%{DATA:traefik.access.backend_url}")?
Resut:
error: field [agent] not present as part of path [traefik.access.agent]
From what I understand is that current pattern is not working properly its not parsing data since user_agent field.
here is example of pattern that works for me:
initial (pattern for support of old version of traefik):
%{IPORHOST:traefik.access.remote_ip} - (?:-|"%{DATA:traefik.access.user_name}") \[%{HTTPDATE:traefik.access.time}\] "%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP/%{NUMBER:traefik.access.http_version}" (?:-|%{NUMBER:traefik.access.response_code:int}) (?:-|%{NUMBER:traefik.access.body_sent.bytes:int}) "(?:-|%{DATA:traefik.access.referrer})" "(?:-|%{DATA:traefik.access.agent})" (?:-|%{NUMBER:traefik.access.request_count:int}) (?:-|"%{DATA:traefik.access.frontend_name}") (?:-|"%{DATA:traefik.access.backend_url}") %{NUMBER:traefik.access.duration:int}ms
fallback (pattern for support of new version of traefik):
%{IPORHOST:traefik.access.remote_ip} - (?:-|"%{DATA:traefik.access.user_name}") \[%{HTTPDATE:traefik.access.time}\] "%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP/%{NUMBER:traefik.access.http_version}" (?:-|%{NUMBER:traefik.access.response_code:int}) (?:-|%{NUMBER:traefik.access.body_sent.bytes:int}) (?:-|"%{DATA:traefik.access.referrer}") (?:-|"%{DATA:traefik.access.agent}") (?:-|%{NUMBER:traefik.access.request_count:int}) (?:-|"%{DATA:traefik.access.frontend_name}") (?:-|"%{DATA:traefik.access.backend_url}") %{NUMBER:traefik.access.duration:int}ms
patterns are based on: https://github.com/containous/traefik/blob/5140bbe99a79b45f98c27fbb8e9b6833194af4cb/middlewares/accesslog/logger_formatters.go
related discussion: https://discuss.elastic.co/t/traefik-module-for-filbeat-dont-parse-traefik-logs/116170/3
Pipeline that works for me
pipeline.json
{
"description": "Pipeline for parsing Traefik access logs. Requires the geoip and user_agent plugins.",
"processors": [{
"grok": {
"field": "message",
"patterns":[
"%{IPORHOST:traefik.access.remote_ip} - (?:-|\"%{DATA:traefik.access.user_name}\") \\[%{HTTPDATE:traefik.access.time}\\] \"%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP/%{NUMBER:traefik.access.http_version}\" (?:-|%{NUMBER:traefik.access.response_code:int}) (?:-|%{NUMBER:traefik.access.body_sent.bytes:int}) \"(?:-|%{DATA:traefik.access.referrer})\" \"(?:-|%{DATA:traefik.access.agent})\" (?:-|%{NUMBER:traefik.access.request_count:int}) (?:-|\"%{DATA:traefik.access.frontend_name}\") (?:-|\"%{DATA:traefik.access.backend_url}\") %{NUMBER:traefik.access.duration:int}ms",
"%{IPORHOST:traefik.access.remote_ip} - (?:-|\"%{DATA:traefik.access.user_name}\") \\[%{HTTPDATE:traefik.access.time}\\] \"%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP/%{NUMBER:traefik.access.http_version}\" (?:-|%{NUMBER:traefik.access.response_code:int}) (?:-|%{NUMBER:traefik.access.body_sent.bytes:int}) (?:-|\"%{DATA:traefik.access.referrer}\") (?:-|\"%{DATA:traefik.access.agent}\") (?:-|%{NUMBER:traefik.access.request_count:int}) (?:-|\"%{DATA:traefik.access.frontend_name}\") (?:-|\"%{DATA:traefik.access.backend_url}\") %{NUMBER:traefik.access.duration:int}ms"
],
"ignore_missing": true
}
},{
"remove":{
"field": "message",
"ignore_failure": true
}
}, {
"rename": {
"field": "@timestamp",
"target_field": "read_timestamp"
}
}, {
"date": {
"field": "traefik.access.time",
"target_field": "@timestamp",
"formats": ["dd/MMM/YYYY:H:m:s Z"]
}
}, {
"remove": {
"field": "traefik.access.time",
"ignore_failure": true
}
}, {
"user_agent": {
"field": "traefik.access.agent",
"target_field": "traefik.access.user_agent",
"ignore_failure": true
}
}, {
"remove": {
"field": "traefik.access.agent",
"ignore_failure": true
}
}, {
"geoip": {
"field": "traefik.access.remote_ip",
"target_field": "traefik.access.geoip"
}
}],
"on_failure" : [{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}