Filebeat: File specific document type, fields, tags

[oazabir]

We usually host multiple virtual directories in a web server. We need to configure one file beat instance to ship logs of all the virtual directories. However, logs for each file needs to have its own tags, document type and fields. Otherwise they have to be all in one format and gets dumped into one index.

We should have a hierarchical format like this:

path:

• /var/httpd/site1.log
  • input_type: log
  • document_type: apachecommon
  • fields: [ "system": "site1", "group": "apachelog" ]
  • tags: [ "site1" ]
  • index: site1-logstash-YY.MM.dd
• /var/weblogic/site2.log
  • input_type: log
  • document_type: weblogic
  • fields: [ "system": "site2", "group": "weblogiclog" ]
  • tags: [ "site2" ]
  • index: site2-logstash-YY.MM.dd

[oazabir]

Or we could adopt a fluentd style model where each path just gets a tag. Then later part of the configuration file, we define the fields, index, document_type etc attributes for each tag.

For ex,

path: 
   tag1: /var/httpd/site1.log
   tag2: /var/httpd/site2.log


tags:
   tag1: 
          input_type: log
          document_type: apachecommon
          ...
   tag2:
          index: ....
          fields: ....

[ruflin]

You can have different fields per file if you use one prospector per file and define fields for each prospectors. We already have tags per beat. Are there things you cannot map with the fields in the prospector?

[oazabir]

Ah that should do. I did not notice the multiple prospector.

Now just need different tags for different prospector.

[ruflin]

With tags you mean fields? Or is fields not sufficient and you need tags in addition?

[MiguelMoll]

Found this ticket while looking for a possible solution. Hopefully this is a good place. Would be great to have a list of "tags" along with key:value fields per prospector. Such as:

tags: [ "site2", "thistag", "thattag"]

[ruflin]

@MiguelMoll Can you briefly elaborate on how you use tags differently from fields. It is important for me to understand the use case.

[MiguelMoll]

Hard to answer exactly but going by how we've used tags in the past with logstash. It allows another avenue for filtering in Kibana.

If fields weren't strictly key:value where the value has to be a string also a list that would work too.

fields:
    level: debug
    tags: [ "site2", "thistag", "thattag"]

With fields_under_root and we'd be good to go.

The shipper uses tags nicely. But would be great to to make some tags prospector specific.

[ruflin]

If you put tags under fields this should already be possible in the upcoming 1.1 thanks to @magnusbaeck #506

[MiguelMoll]

Excellent news! Hopefully I didn't derail this ticket too much.

[ruflin]

The question is if the above would also solve the problem from @oazabir ?

@MiguelMoll Feel free to try the snapshots for 1.1. Links can be found in this post: https://discuss.elastic.co/t/insufficient-throughput-from-filebeat/39564/4

[ruflin]

Here is the generic tag / fields implementation @andrewkroh did: #1092 I think this resolves the above issues. Closing this issue, but happy to discuss further.

[sunilmchaudhari]

Hi,
How can I create indexes on the basis of fields created in FB?
fields:
level: debug
tags: [ "site2", "thistag", "thattag"]
Application: "myapp"

So the index should be created as myap-YYYY-MM-DD