[Filebeat] Select output index based on the source input #14010

faec · 2019-10-10T21:22:38Z

Adds configuration to Filebeat inputs to override the output index of their events (#13255)

…input.index config

urso · 2019-10-11T12:44:14Z

Passing a fmtstr via Meta indeed is hacky. Meta should be treated as a document itself, only storing serializable/final values. We send Meta as @metadata to Kafka/Logstash, so users can mimic the behavior Beats->ES in the Beats->LS->ES scenario as well.

About Wrapper. The pipelineConnector in filebeat/channel/connector.go already is a wrapper for the beat.Pipeline. It reimplements ConnectWith, adjusting user settings so to apply common settings. E.g. fields, tags, processors.

The pipelineConnector creates a list of processors, depending on user settings. Next libbeat merges the local processors with global ones. The final processors pipeline is established here: https://github.com/elastic/beats/blob/master/libbeat/publisher/processing/default.go#L223

I think as of now building the processor pipeline is much too complex, and something I'd like to clean up in the future. Settings like fields, tags should be replaced by processors.

But having processors capabilities, I'm thinking to have the pipelineConnector add an index-name-processor if "index" is configured. In this case the index name will be created by the processors and passed along as plain string to each output.

The only limitation would be that global processors follow global processors. Users configuring parsing on a global level might run into issues (but we don't support event fields anyways, and we do not parse timestamps). This is indeed some technical debt, that I hope we can reduce by simplifying the processor setup overall.

Just thinking loud (no need to implement this):
A more generic approach would be a "format" processor (e.g. kafka/file/console output have a format codec), that can create a string based on the event contents. In this case the format.target: "@metadata.index" could be used to configure the target index.

urso · 2019-10-11T12:45:43Z

libbeat/idxmgmt/std.go

+// This helper mimicks applyStaticFmtstr in ilm.go, creating a placeholder
+// event for the restricted set of fields we allow here. It might be worth
+// making this a shared helper function or otherwise specializing for this case.
+func expandIndexPattern(


We have this pattern in a few places. When 7.0 was in the making, devs did forget to adapt one or the other place. Maybe we can provide a common formatter type/instance that can provide these default fields?

filebeat/channel/connector.go

urso · 2019-10-25T15:46:17Z

libbeat/common/fmtstr/formattimestamp.go

+type TimestampFormatString struct {
+	eventFormatString *EventFormatString
+	fields            common.MapStr
+}


urso · 2019-10-25T15:49:44Z

libbeat/idxmgmt/std.go

+		if idx, ok := tmp.(string); ok {
+			return idx
+		}
+	}


hm... where is alias used? The name looks very specific to some use-case. I guess it was added to work around the timestamp addition when index is used. Should we go with 'alias' or 'raw-index' in general?

I don't know who might be using alias or how, which is why I was reluctant to overload it (that, and because of the potentially confusing name)... my preference is for raw-index or something similar (raw-index-name?) since the functional behavior is independent of whether aliases are being used, though I could be convinced to go with alias instead to avoid proliferating magic Metadata fields...

Looks like we need to grep the source. I guess this comes from Metricbeat or Heartbeat.

this is odd I don't remember that?

I think I copied it from the original ILM PR. It was introduced in 7.0. Not sure if anyone still uses it. Removing could break things. We should at least create a follow up issue (technical debt) to investigate and remove it in the future.

+1 on not relying on alias here.

faec · 2019-10-29T19:56:15Z

Finished the last pieces including new unit tests confirming the order of application of all the processor settings. I think the only question left is whether to keep the metadata name "raw-index" (or "raw_index"? I'm not sure which is more in line with our conventions), or overload the "alias" field.

filebeat/channel/connector.go

libbeat/common/fmtstr/formattimestamp_test.go

urso · 2019-10-30T02:00:02Z

I think the only question left is whether to keep the metadata name "raw-index" (or "raw_index"? I'm not sure which is more in line with our conventions), or overload the "alias" field.

Let's not overload "alias". We actually have to figure if it is in use, or some debt.

Everything in Meta should be treated like normal events. We even send Meta to Logstash and others (@metadata namespace).

For events we normally prefer to use namespaces. For example @metadata.index.prefix and @metadata.index.name might be appropriate names. Unfortuntaly index is gone and it's functionality doesn't reflect the actual name.

In case ECS does not create a namespace it uses _ to separate words. To be consistent with ECS we should use raw_index.

faec · 2019-10-30T14:55:27Z

Ok, all comments addressed, and docs + changelog updated, so I think I'm just waiting on CI now

urso · 2019-10-30T15:58:47Z

Looks like the intake job doesn't like the formatting of libbeat/common/fmtstr/formattimestamp_test.go.

(cherry picked from commit 6a03478)

…e source input (#14664) * [Filebeat] Select output index based on the source input (#14010) (cherry picked from commit 6a03478) * Delete debug file, fix changelog merge

(cherry picked from commit 6a03478)

…e source input (#15094)

…d on the source input (elastic#14664) * [Filebeat] Select output index based on the source input (elastic#14010) (cherry picked from commit 82c0bde) * Delete debug file, fix changelog merge

faec added 2 commits October 9, 2019 16:28

Define wrapped Connectors and Outleters and use them to pass through …

e243d96

…input.index config

Add string formatting to the input index field

de02aeb

faec requested a review from urso October 10, 2019 21:22

urso reviewed Oct 11, 2019

View reviewed changes

dedemorton mentioned this pull request Oct 16, 2019

7.5 Beats documentation roadmap #14081

Closed

6 tasks

faec added 6 commits October 18, 2019 13:06

Replace custom wrapped channels with processors

0515696

Add beat.Info to OutletFactory

e510493

Clean up minimal event string formatting + move to shared location

ac46926

StaticEventFormatString -> TimestampFormatString

40ae68c

Clean up the api for TimestampFormatString

2d5da87

Working on pipelineConnector unit test

0042aaf

urso approved these changes Oct 25, 2019

View reviewed changes

Factor out processor list construction, finish unit tests

e9b01e9

faec changed the title ~~[WIP] [Filebeat] Select output index based on the source input~~ [Filebeat] Select output index based on the source input Oct 29, 2019

urso reviewed Oct 30, 2019

View reviewed changes

filebeat/channel/connector.go Outdated Show resolved Hide resolved

urso reviewed Oct 30, 2019

View reviewed changes

libbeat/common/fmtstr/formattimestamp_test.go Show resolved Hide resolved

faec added 4 commits October 30, 2019 10:14

Address review comments

5d735a9

Document the new option

d255d44

Merge branch 'master' into input-index

372348c

Update changelog

5badc14

urso approved these changes Oct 30, 2019

View reviewed changes

ph approved these changes Oct 30, 2019

View reviewed changes

faec added 3 commits October 30, 2019 12:10

regenerate formatting

beac274

update *more* formatting

486bcc9

Use the right arguments in filebeat beater?

5faafb1

faec added 2 commits November 7, 2019 14:23

remove vestigial backticks in comment

406c09e

Merge branch 'master' into input-index

34455c8

faec added Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. labels Nov 7, 2019

faec merged commit 6a03478 into elastic:master Nov 7, 2019

faec deleted the input-index branch November 7, 2019 21:38

faec added a commit to faec/beats that referenced this pull request Nov 20, 2019

[Filebeat] Select output index based on the source input (elastic#14010)

9d64b3d

(cherry picked from commit 6a03478)

faec added a commit to faec/beats that referenced this pull request Nov 20, 2019

[Filebeat] Select output index based on the source input (elastic#14010)

2926f20

(cherry picked from commit 6a03478)

faec mentioned this pull request Nov 20, 2019

Cherry-pick #14010 to 7.5: [Filebeat] Select output index based on the source input #14664

Merged

faec added v7.5.0 and removed needs_backport PR is waiting to be backported to other branches. labels Nov 20, 2019

faec added a commit to faec/beats that referenced this pull request Nov 20, 2019

[Filebeat] Select output index based on the source input (elastic#14010)

be728cc

(cherry picked from commit 6a03478)

faec mentioned this pull request Nov 20, 2019

[filebeat] Revert accidental checkin of a local debug config file #14665

Merged

faec mentioned this pull request Dec 3, 2019

[metricbeat] Allow inputs to specify output index #14923

Closed

This was referenced Dec 11, 2019

[Winlogbeat] Allow inputs to specify output index #15062

Closed

[Journalbeat] Allow inputs to specify output index #15063

Closed

[Functionbeat] Allow inputs to specify output index #15064

Closed

[Auditbeat] Allow inputs to specify output index #15065

Closed

ycombinator mentioned this pull request Dec 13, 2019

Journalbeat: add index option to input #15071

Merged

faec mentioned this pull request Dec 13, 2019

Cherry-pick #14010 to 7.x: [Filebeat] Select output index based on the source input #15094

Merged

faec added the v7.6.0 label Dec 13, 2019

faec added a commit to faec/beats that referenced this pull request Dec 13, 2019

[Filebeat] Select output index based on the source input (elastic#14010)

fc43b80

(cherry picked from commit 6a03478)

ycombinator mentioned this pull request Dec 13, 2019

Functionbeat: add index option to function configuration #15101

Merged

faec added a commit that referenced this pull request Dec 13, 2019

Cherry-pick #14010 to 7.x: [Filebeat] Select output index based on th…

1bbac9f

…e source input (#15094)

dedemorton mentioned this pull request Jan 15, 2020

7.6.0 Observability Documentation #15595

Closed

14 tasks

jorgemarey pushed a commit to jorgemarey/beats that referenced this pull request Jun 8, 2020

[Filebeat] Select output index based on the source input (elastic#14010)

aceeded

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Select output index based on the source input #14010

[Filebeat] Select output index based on the source input #14010

faec commented Oct 10, 2019

urso commented Oct 11, 2019

urso Oct 11, 2019

urso Oct 25, 2019

urso Oct 25, 2019

faec Oct 25, 2019

urso Oct 25, 2019

ph Oct 28, 2019

urso Oct 30, 2019 •

edited

Loading

faec commented Oct 29, 2019

urso commented Oct 30, 2019

faec commented Oct 30, 2019

urso commented Oct 30, 2019

[Filebeat] Select output index based on the source input #14010

[Filebeat] Select output index based on the source input #14010

Conversation

faec commented Oct 10, 2019

urso commented Oct 11, 2019

urso Oct 11, 2019

Choose a reason for hiding this comment

urso Oct 25, 2019

Choose a reason for hiding this comment

urso Oct 25, 2019

Choose a reason for hiding this comment

faec Oct 25, 2019

Choose a reason for hiding this comment

urso Oct 25, 2019

Choose a reason for hiding this comment

ph Oct 28, 2019

Choose a reason for hiding this comment

urso Oct 30, 2019 • edited Loading

Choose a reason for hiding this comment

faec commented Oct 29, 2019

urso commented Oct 30, 2019

faec commented Oct 30, 2019

urso commented Oct 30, 2019

urso Oct 30, 2019 •

edited

Loading