Closed
Description
Since moving beats from indices to datastreams with version 8.x the setup command does not properly prepare index-management. The setup task creates the required index templates, but does not bootstrap the datastream. This leads to errors when a user with minimal privileges is used for indexing.
How to recreate:
- Use freshly installed elasticsearch cluster
- Run
filebeat setupwith userelastic - Set up user publishing user as per documentation
- Run filebeat with publishing user
User:
{
"filebeat_publisher" : {
"username" : "filebeat_publisher",
"roles" : [
"filebeat_publisher"
],
"full_name" : "",
"email" : "",
"metadata" : { },
"enabled" : true
}
}
Role:
{
"filebeat_publisher" : {
"cluster" : [
"monitor",
"read_ilm",
"read_pipeline"
],
"indices" : [
{
"names" : [
"filebeat-*"
],
"privileges" : [
"create_doc",
"view_index_metadata"
],
"allow_restricted_indices" : false
}
],
"applications" : [ ],
"run_as" : [ ],
"metadata" : { },
"transient_metadata" : {
"enabled" : true
}
}
}
Setup with elastic user:
root@es1:/etc/filebeat# filebeat setup
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded Ingest pipelines
Template available after setup:
{
"index_templates" : [
{
"name" : "filebeat-8.0.1",
"index_template" : {
"index_patterns" : [
"filebeat-8.0.1"
[...]
API show that Datastream is not yet available GET _data_stream/filebeat*:
{
"data_streams" : [ ]
}
filebeat test output with user filebeat_publisher:
root@es1:/etc/filebeat# filebeat test output
elasticsearch: https://localhost:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 127.0.0.1
dial up... OK
TLS...
security... WARN server's certificate chain verification is disabled
handshake... OK
TLS version: TLSv1.3
dial up... OK
talk to server... OK
version: 8.0.1
filebeat error when running with user filebeat_publisher:
{
"log.level": "warn",
"@timestamp": "2022-03-02T16:57:07.875Z",
"log.logger": "elasticsearch",
"log.origin": {
"file.name": "elasticsearch/client.go",
"file.line": 414
},
"message": "Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.March, 2, 16, 57, 5, 119819138, time.Local), Meta:{\"pipeline\":\"filebeat-8.0.1-system-syslog-pipeline\"}, Fields:{\"agent\":{\"ephemeral_id\":\"cb6458df-2050-40fc-aa65-1e925ccba916\",\"id\":\"53c8c63a-a2c9-42f4-ae37-963e590d2068\",\"name\":\"es1\",\"type\":\"filebeat\",\"version\":\"8.0.1\"},\"ecs\":{\"version\":\"1.12.0\"},\"event\":{\"dataset\":\"system.syslog\",\"module\":\"system\",\"timezone\":\"+00:00\"},\"fileset\":{\"name\":\"syslog\"},\"host\":{\"architecture\":\"x86_64\",\"containerized\":false,\"hostname\":\"es1\",\"id\":\"3959ee78599046eca2cb48b212958d85\",\"ip\":[\"10.0.2.15\",\"fe80::f6:7fff:fe02:5c72\",\"192.168.56.31\",\"fe80::a00:27ff:fedf:ab10\"],\"mac\":[\"02:f6:7f:02:5c:72\",\"08:00:27:df:ab:10\"],\"name\":\"es1\",\"os\":{\"codename\":\"focal\",\"family\":\"debian\",\"kernel\":\"5.4.0-90-generic\",\"name\":\"Ubuntu\",\"platform\":\"ubuntu\",\"type\":\"linux\",\"version\":\"20.04.3 LTS (Focal Fossa)\"}},\"input\":{\"type\":\"log\"},\"log\":{\"file\":{\"path\":\"/var/log/syslog\"},\"offset\":108939},\"message\":\"Mar 2 15:03:33 ubuntu-focal multipathd[494]: uevent trigger error\",\"service\":{\"type\":\"system\"}}, Private:file.State{Id:\"native::70979-2049\", PrevId:\"\", Finished:false, Fileinfo:(*os.fileStat)(0xc0007e1450), Source:\"/var/log/syslog\", Offset:109006, Timestamp:time.Date(2022, time.March, 2, 16, 57, 5, 38485206, time.Local), TTL:-1, Type:\"log\", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x11543, Device:0x801}, IdentifierName:\"native\"}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=403): {\"type\":\"security_exception\",\"reason\":\"action [indices:admin/auto_create] is unauthorized for user [filebeat_publisher] with roles [filebeat_publisher], this action is granted by the index privileges [auto_configure,create_index,manage,all]\"}, dropping event!",
"service.name": "filebeat",
"ecs.version": "1.6.0"
}
Heres the interesting part of the error:
\"reason\":\"action [indices:admin/auto_create] is unauthorized for user [filebeat_publisher] with roles [filebeat_publisher], this action is granted by the index privileges [auto_configure,create_index,manage,all]\"}, dropping event!"
Starting filebeat with user elastic bootstraps the datastream:
{
"data_streams" : [
{
"name" : "filebeat-8.0.1",
"timestamp_field" : {
"name" : "@timestamp"
},
"indices" : [
{
"index_name" : ".ds-filebeat-8.0.1-2022.03.02-000001",
"index_uuid" : "-clfw8CNQp-NOsxqquPR2w"
}
],
"generation" : 1,
"status" : "YELLOW",
"template" : "filebeat-8.0.1",
"ilm_policy" : "filebeat",
"hidden" : false,
"system" : false,
"allow_custom_routing" : false,
"replicated" : false
}
]
}
After that, running filebeat with the publishing user works as intended.
IMO the setup --index-management should create the datastream.