Closed
Description
While running the Filebeat googlecloud module we noticed an error messaging regarding events failing to be indexed to Elasticsearch in the logs:
2020-05-12T15:01:14.954Z WARN elasticsearch/client.go:517 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x3a3bab48, ext:63724892449, loc:(*time.Location)(nil)}, Meta:{"id":"d0bfda02f5-1183688770061909","pipeline":"filebeat-7.6.2-googlecloud-audit-pipeline"}, Fields:{"agent":{"ephemeral_id":"afa01cad-adbd-45cf-98b4-507d4eb62191","hostname":"filebeat-gcp-statefulset-0","id":"2c6e66ca-0d25-430d-b261-da3911e1cab9","type":"filebeat","version":"7.6.2"},"cloud":{"project":{"id":"gcp-project"}},"ecs":{"version":"1.4.0"},"event":{"created":"2020-05-12T15:01:14.049Z","dataset":"googlecloud.audit","id":"d0bfda02f5-1183688770061909","module":"googlecloud","original":"{\"insertId\":\"f1440e02-c072-43b3-a1ef-14699d457c01\",\"labels\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:controller:cronjob-controller\\\" of ClusterRole \\\"system:controller:cronjob-controller\\\" to ServiceAccount \\\"cronjob-controller/kube-system\\\"\"},\"logName\":\"projects/gcp-project/logs/cloudaudit.googleapis.com%2Factivity\",\"operation\":{\"first\":true,\"id\":\"f1440e02-c072-43b3-a1ef-14699d457c01\",\"last\":true,\"producer\":\"k8s.io\"},\"protoPayload\":{\"@type\":\"type.googleapis.com/google.cloud.audit.AuditLog\",\"authenticationInfo\":{\"principalEmail\":\"system:serviceaccount:kube-system:cronjob-controller\"},\"authorizationInfo\":[{\"granted\":true,\"permission\":\"io.k8s.batch.v1.jobs.delete\",\"resource\":\"batch/v1/namespaces/default/jobs/gsuite-exporter-1589294700\"}],\"methodName\":\"io.k8s.batch.v1.jobs.delete\",\"request\":{\"@type\":\"batch.k8s.io/v1.DeleteOptions\",\"apiVersion\":\"batch/v1\",\"kind\":\"DeleteOptions\",\"propagationPolicy\":\"Background\"},\"requestMetadata\":{\"callerIp\":\"::1\",\"callerSuppliedUserAgent\":\"kube-controller-manager/v1.14.10 (linux/amd64) kubernetes/145f9e2/system:serviceaccount:kube-system:cronjob-controller\"},\"resourceName\":\"batch/v1/namespaces/default/jobs/gsuite-exporter-1589294700\",\"response\":{\"@type\":\"core.k8s.io/v1.Status\",\"apiVersion\":\"v1\",\"details\":{\"group\":\"batch\",\"kind\":\"jobs\",\"name\":\"gsuite-exporter-1589294700\",\"uid\":\"2beff34a-945f-11ea-bacf-42010a80007f\"},\"kind\":\"Status\",\"metadata\":{},\"status\":\"Success\"},\"serviceName\":\"k8s.io\",\"status\":{}},\"receiveTimestamp\":\"2020-05-12T15:01:13.578169149Z\",\"resource\":{\"labels\":{\"cluster_name\":\"k8s-cluster\",\"location\":\"us-central1-a\",\"project_id\":\"gcp-project\"},\"type\":\"k8s_cluster\"},\"timestamp\":\"2020-05-12T15:00:49.976989Z\"}","outcome":"success"},"fileset":{"name":"audit"},"googlecloud":{"audit":{"authentication_info":{"principal_email":"system:serviceaccount:kube-system:cronjob-controller"},"authorization_info":[{"granted":true,"permission":"io.k8s.batch.v1.jobs.delete","resource":"batch/v1/namespaces/default/jobs/gsuite-exporter-1589294700","resource_attributes":null}],"method_name":"io.k8s.batch.v1.jobs.delete","request":{"apiVersion":"batch/v1","kind":"DeleteOptions","propagationPolicy":"Background","proto_name":"batch.k8s.io/v1.DeleteOptions"},"request_metadata":{"caller_ip":"::1","caller_supplied_user_agent":"kube-controller-manager/v1.14.10 (linux/amd64) kubernetes/145f9e2/system:serviceaccount:kube-system:cronjob-controller"},"resource_name":"batch/v1/namespaces/default/jobs/gsuite-exporter-1589294700","response":{"@type":"core.k8s.io/v1.Status","apiVersion":"v1","details":{"group":"batch","kind":"jobs","name":"gsuite-exporter-1589294700","uid":"2beff34a-945f-11ea-bacf-42010a80007f"},"kind":"Status","metadata":{},"status":"Success"},"service_name":"k8s.io","status":{},"type":"type.googleapis.com/google.cloud.audit.AuditLog"}},"host":{"name":"filebeat-gcp-statefulset-0"},"input":{"type":"google-pubsub"},"labels":{"logging.googleapis.com/timestamp":"2020-05-12T15:00:49.976989Z"},"log":{"logger":"projects/gcp-project/logs/cloudaudit.googleapis.com%2Factivity"},"service":{"name":"k8s.io","type":"googlecloud"},"source":{"ip":"::1"},"user":{"email":"system:serviceaccount:kube-system:cronjob-controller"},"user_agent":{"original":"kube-controller-manager/v1.14.10 (linux/amd64) kubernetes/145f9e2/system:serviceaccount:kube-system:cronjob-controller"}}, Private:(*pubsub.Message)(0xc000a57170), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [googlecloud.audit.response.status] tried to parse field [status] as object, but found a concrete value"}
Filebeat templates were installed prior to indexing data and have not been modified.
Configuration:
filebeat.yml: |-
filebeat.inputs:
# Templates are installed via setup job in Epsilon
setup.template.enabled: false
filebeat.modules:
- module: googlecloud
audit:
enabled: true
var.project_id: gcp-project
var.topic: gcp-audit-logs
var.subscription_name: audit-log-subscription
var.credentials_file: /usr/share/filebeat/secrets/credentials.json
var.keep_original_message: true
firewall:
enabled: true
var.project_id: gcp-project
var.topic: gcp-firewall-logs
var.subscription_name: firewall-log-subscription
var.credentials_file: /usr/share/filebeat/secrets/credentials.json
var.keep_original_message: true
vpcflow:
enabled: true
var.project_id: gcp-project
var.topic: gcp-vpcflow-logs
var.subscription_name: vpcflow-log-subscription
var.credentials_file: /usr/share/filebeat/secrets/credentials.json
var.keep_original_message: true
output.elasticsearch:
hosts: 'https://${ELASTICSEARCH_ADDRESS}'
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
ssl:
enabled: true
supported_protocols: TLSv1.2
For confirmed bugs, please report:
- Version: 7.6.2
- Operating System: GKE CoreOS