Add Netflow integration 0.0.2 #338

alakahakai · 2020-04-12T01:12:52Z

This PR adds the Netflow integration as a replacement of the Filebeat netflow module.

Here are some screenshots of the ingest manager UI.

ruflin

This looks like a great start. To make this easier to review and faster to iterate I suggest the following: Open 2 separate PR's for cisco and netflow. And for each, lets get a very basic version in first, meaning for example only have a single dataset inside that only supports one input and then start iterating on it? This should allow us to move quicker.

dev/packages/example/cisco-1.0.0/dataset/asa/filebeat/config/input.yml

dev/packages/example/cisco-1.0.0/dataset/asa/filebeat/module.yml

dev/packages/example/cisco-1.0.0/dataset/asa/agent/input/input.yml

dev/packages/example/cisco-1.0.0/dataset/ftd/agent/input/input.yml

dev/packages/example/cisco-1.0.0/manifest.yml

dev/packages/example/netflow-1.0.0/dataset/log/agent/input/input.yml

mtojek · 2020-04-14T08:09:48Z

Did you use the import-beats script?

alakahakai · 2020-04-17T19:08:00Z

Did you use the import-beats script?

Yes, but I had to make changes according to the coredns package, which @ruflin asked me to follow. I will give it a try end-to-end once the fleet function is working in ingest manager. It was crashing Kibana when I tried it last time.

mtojek · 2020-04-20T07:18:13Z

Did you use the import-beats script?

Yes, but I had to make changes according to the coredns package, which @ruflin asked me to follow. I will give it a try end-to-end once the fleet function is working in ingest manager. It was crashing Kibana when I tried it last time.

When you notice that it crashed or generated dashboards in a wrong way, please open an issue and bring samples. It will help improve the generating process.

andrewkroh · 2020-04-20T14:59:24Z

dev/packages/example/cisco-1.0.0/dataset/asa/agent/input/input.yml

@@ -0,0 +1,14 @@
+{{#if input == "syslog"}}


What's the templating language used in these files?

@ruflin I think it would be nice to use the appropriate file extension for the template type (e.g. .erb, .j2, .mustache) to make it clear to editors (both the people and the tools).

It is handlebars (reminds me off https://github.com/elastic/package-registry/issues/179). And the above is actually not valid handlebars :-( Having support for 2 inputs is coming here: #346

I agree we should probably adjust the file extension to make it better for the tooling. If we do, I wonder if we should the same for ingest pipeline files which reference an other pipeline. The challenge I see here is that 99% of the pipelines don't use it but a very few do. Does this still mean all should have to have a handlebars suffix or do we just check for both files?

I think it would be easier if files that use templating had an extension that indicates it. Then when writing the code that uses the modules you know when you need to preprocess the file, and then you don't need to guess or unnecessarily preprocess every file like we do today in Beats.

I wonder if supporting both types and not process would complicate the code? What is your concern about processing files that don't need processing?

My concern is that you need to make sure you haven't used content that requires escaping in the templating language. So if you don't need those features you just call the file config.yml instead of config.yml.handlebars. Hopefully that isn't too much of issue in most files. Maybe we just process all files through the template engine for now, but put the template extension on them now.

I'd just be happy if my editor could easily recognize that these are template type X and recommend the appropriate plugin. 😄

Should we go with config.yml.hbs or config.hbs? I skiped handlebars as it is very long and .hbs seems to work at least in my IDE too. I kind of like the two suffix so the end goal format is also in there.

config.yml.hbs SGTM

@ruflin if this is an update to the format of package, please open an issue in the repository for adjusting all packages.

I introduced the .yml.hbs naming in this PR: #380

@mtojek thanks for taking care of it. I assume this make the issue obsolete.

alakahakai · 2020-04-20T21:19:45Z

Made this for netflow only. Will open another one for cisco.

ruflin · 2020-04-21T07:04:31Z

@alakahakai Let me know when it's ready for review.

…y into netflow_cisco

mtojek

Screenshots in this PR would be welcome :)

ruflin · 2020-04-28T08:01:40Z

dev/packages/alpha/netflow-0.0.2/dataset/log/agent/stream/stream.yml

+max_message_size: '{{this.max_message_size}}'
+expiration_timeout: '{{this.expiration_timeout}}'
+queue_size: {{this.queue_size}}
+{{#if this.timeout}}


I'm stumbling over the this here. @mtojek do you know if this is necessary?

This is already fixed, even though both forms (with/without this) are correct.

The previous implementation of processing stream configuration left this for consistency. If you upgrade to the new implementation, you will see a better formed file: https://github.com/elastic/package-registry/blob/master/dev/packages/beats/netflow-0.0.1/dataset/log/agent/stream/netflow.yml.hbs

ruflin

As soon as this package landed, that dashboards should be revisted. My assumption is that with the new indexing strategy a lot of queries can be made more efficient by targeting much fewer indices.

ruflin · 2020-04-28T08:02:25Z

dev/packages/alpha/netflow-0.0.2/dataset/log/fields/ecs.yml

@@ -0,0 +1,3141 @@
+- name: '@timestamp'


Does this only contain the fields that are used by this dataset or is this all of ECS? The goal should be to only have the fields in here that are actually used.

So using the netflow module as an example the fields consist of:

some base filebeat fields that may not be in ECS (like input.type)

all filebeat netflow input fields (some 400+ fields)

a subset of ECS namespaces (like agent, source, destination, network, event)

Very few of these fields are something that the module actually creates. This module basically brings together Filebeat with an Ingest Node pipeline to add geo location and autonomous system numbers. So keeping this module updated will be a lot of work. And since it depends on Filebeat's fields, this template will vary based on the Filebeat version.

So I wonder if it would be possible to list some dependent templates in addition to custom fields created by a module to create a composite. Like saying that for this module it depends on

filebeat $automatic_version | include_namespaces(agent | netflow | input.type | source | destination)

ECS 1.4 | include_namespaces(source | destination)

@andrewkroh Nice diagram!

v2 Elasticsearch templates support components and one of the reasons we pushed for this is exactly the above problem. So long term, I'm pretty sure we will support this. Short term, I think we should get this into the development process. The current plan is to use the integrations repository for the development of the packages. There you could even use sysmlinks or whatever reference you need to in the end build the final package with all the files you need for the above. This keeps the final package simple for now.

I was not aware that netflow has 400+ fields (sounds like a lot) so in this case, the template will be pretty big in any case. Good news it is only the netflow template and does not affect anything else. Also it shouldn't be a problem if we have too many fields right now and later optimise as during update, the template will be overwritten.

Yeah, I think some symlinks could accomplish this. Does the current module more allow for more than one fields file?

you can have as many as you want as long as they end in .yml and are in the correct directory.

dev/packages/alpha/netflow-0.0.2/docs/README.md

dev/packages/alpha/netflow-0.0.2/kibana/dashboard/34e26884-161a-4448-9556-43b5bf2f62a2.json

elasticmachine · 2020-05-08T08:53:52Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Branch indexing]
Start Time: 2020-05-19T12:18:21.151+0000
Duration: 4 min 16 sec (255933)

mtojek

Please add some screenshots presenting the Kibana UI.

…y into netflow_cisco

mtojek · 2020-05-15T09:08:11Z

Here are some spottings on my side:

NetFlow log logs: maybe NetFlow standard logs?
netflow_host, netflow_port: the prefix netflow is redundant here
all configuration variables: UI labels should be human readable and not contain "_" (underscore)
not sure if we need to show to user all variables - maybe just host and port?
all configuration variables expect host and port: it would be feasible to leave a one sentence description
all configuration variables should contain information about their types (text, number, etc.)

mtojek

Not blocking, let's iterate on top of this.

Add example packages: netflow-1.0.0, cisco-1.0.0

b58158e

alakahakai added the review label Apr 12, 2020

alakahakai requested a review from ruflin April 12, 2020 02:42

andrewkroh self-requested a review April 13, 2020 15:08

ruflin reviewed Apr 14, 2020

View reviewed changes

ruflin assigned alakahakai Apr 15, 2020

andrewkroh reviewed Apr 20, 2020

View reviewed changes

Make this PR only for netflow example package.

343833f

alakahakai changed the title ~~Add example packages: netflow-1.0.0, cisco-1.0.0~~ Add example packages: netflow-1.0.0 Apr 20, 2020

Lei Qiu added 3 commits April 21, 2020 09:51

Add example packages: netflow-1.0.0, cisco-1.0.0

dbd7809

Make this PR only for netflow example package.

01305d7

Merge branch 'netflow_cisco' of github.com:alakahakai/package-registr…

1dfd2bb

…y into netflow_cisco

alakahakai changed the title ~~Add example packages: netflow-1.0.0~~ Add example packages: netflow Apr 22, 2020

alakahakai changed the title ~~Add example packages: netflow~~ Add alpha package: netflow Apr 22, 2020

alakahakai changed the title ~~Add alpha package: netflow~~ Add netflow integration Apr 23, 2020

alakahakai changed the title ~~Add netflow integration~~ Add Netflow integration Apr 23, 2020

alakahakai marked this pull request as draft April 23, 2020 20:27

alakahakai changed the title ~~Add Netflow integration~~ Add Netflow integration 0.0.2 Apr 23, 2020

alakahakai requested review from mtojek and ruflin April 23, 2020 20:28

mtojek reviewed Apr 27, 2020

View reviewed changes

ruflin reviewed Apr 28, 2020

View reviewed changes

Update netflow integration

523c9a8

Update netflow integration

6fb3777

alakahakai requested review from ruflin and mtojek May 14, 2020 05:55

mtojek reviewed May 14, 2020

View reviewed changes

Lei Qiu added 3 commits May 14, 2020 22:25

Update manifest.yml

d160c55

Fix manifest.yml

4985b67

Merge branch 'netflow_cisco' of github.com:alakahakai/package-registr…

55248b8

…y into netflow_cisco

alakahakai requested a review from mtojek May 15, 2020 05:32

Ray Qiu added 2 commits May 15, 2020 11:57

Merge remote-tracking branch 'upstream/master' into netflow_cisco

9e424f2

Update netflow integration

87c2918

alakahakai marked this pull request as ready for review May 18, 2020 22:35

mtojek approved these changes May 19, 2020

View reviewed changes

alakahakai merged commit 6fd650c into elastic:master May 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Netflow integration 0.0.2 #338

Add Netflow integration 0.0.2 #338

alakahakai commented Apr 12, 2020 •

edited

Loading

ruflin left a comment

mtojek commented Apr 14, 2020

alakahakai commented Apr 17, 2020

mtojek commented Apr 20, 2020

andrewkroh Apr 20, 2020

ruflin Apr 20, 2020

andrewkroh Apr 20, 2020

ruflin Apr 21, 2020

andrewkroh Apr 22, 2020

ruflin Apr 23, 2020

andrewkroh Apr 23, 2020

mtojek Apr 24, 2020

mtojek Apr 24, 2020

ruflin Apr 27, 2020

alakahakai commented Apr 20, 2020

ruflin commented Apr 21, 2020

mtojek left a comment

ruflin Apr 28, 2020

mtojek Apr 28, 2020

ruflin left a comment

ruflin Apr 28, 2020

andrewkroh Apr 28, 2020

ruflin Apr 28, 2020

andrewkroh Apr 28, 2020

ruflin Apr 28, 2020

elasticmachine commented May 8, 2020 •

edited

Loading

Build stats

mtojek left a comment

mtojek commented May 15, 2020

mtojek left a comment

Add Netflow integration 0.0.2 #338

Add Netflow integration 0.0.2 #338

Conversation

alakahakai commented Apr 12, 2020 • edited Loading

ruflin left a comment

Choose a reason for hiding this comment

mtojek commented Apr 14, 2020

alakahakai commented Apr 17, 2020

mtojek commented Apr 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alakahakai commented Apr 20, 2020

ruflin commented Apr 21, 2020

mtojek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented May 8, 2020 • edited Loading

💚 Build Succeeded

Build stats

mtojek left a comment

Choose a reason for hiding this comment

mtojek commented May 15, 2020

mtojek left a comment

Choose a reason for hiding this comment

alakahakai commented Apr 12, 2020 •

edited

Loading

elasticmachine commented May 8, 2020 •

edited

Loading