Ability to compare if fields are identical from different sources

[Betsworth22]

We are trying to compare ip addresses from our firewall and ip reputation database. This was not an option in logstash and we were hoping we could accomplish this at some point in Kibana.

[rashidkpc]

Can you please describe exactly how you would envision this feature working?

[Betsworth22]

Ok lets just use our example to help simplify how it would work. Lets say you have information coming in from a firewall and it is setup like @type:firewall and within that information you have a src_ip and dst_ip setup as fields. (Examples @fields.src_ip and @fields.dst_ip)
Our second source is coming in from an ip-reputation database known as threats so it is setup as @type:threats with both a @fields.src_ip and @fields.dst_ip.

Compare two fields and filter events that match only. So if a src_ip matches a known threat src_ip it would show up in the logs listed in Kibana and filter out the ones that do not match.
I see the panel options being something like below
Source 1
Type : @type:firewall Field : @fields.src_ip
Source 2
Type : @type:threats Field : @fields.src_ip

All it has to do is search the fields and match only identical results and then filter out the ones that do not. Then I can use the current panels and charts to make it look pretty. Great job on this product by the way. We look forward to helping you grow this product and offer ideas and support where we can.

Cody

[Betsworth22]

Accidentally closed this issue. Reopening.

[rashidkpc]

Unfortunately this would really require join support on the Elasticsearch side. You can track the progress of that here: elastic/elasticsearch#6769

[Bl4ckD4wn]

some information when this issue is going to be fix?

[jccq]

For those interested, we extended Matt Weber original PR elastic/elasticsearch#3278 into a plugin and are maintaining a Kibana version that supports this (relational filters etc) please check http://siren.solution/kibi

it does have limitations, but within those, it works nicely

[kimchy]

Please note, myself and we at Elastic don't consider the Join support built on top of Matt Webber pull request (which ended up being copied by Kibi) to be correct implementation, which is the reason we did not pull it in at the end. The set of limitations it imposes ends up contradicting one of the most fundamental concepts we have in Elasticsearch, which is building features that scale.

@jccq I will mention this position in other threads, since there seem to be a flux of comments on several other GitHub issues and threads here, and I would appreciate slowing down on those.