Top level: "client" and "server"

[ave19]

Hey! We do a lot of network flow work. We have a sort of issue using "source" and "destination" because flow data comes in both directions and we get records for each. The data for a single session might look like:

source.ip	source.port	destination.ip	desatination.port
1.2.3.4	54321	6.7.8.9	443
6.7.8.9	443	1.2.3.4	5432

So that's a problem for us. The concepts of source and destination really only apply on a packet scale anyway. We'd like to normalize both of the records into:

client.ip	client.port	server.ip	server.port
1.2.3.4	54321	6.7.8.9	443

This would also sort through things like DNS requests and other services that open a port.

Thoughts about that?

[webmat]

Indeed, we've been pondering that already, after @robcowart's comment in this thread.

I wonder with this strategy, which side is being called "server", in cases where the infrastructure under management is calling to the outside (e.g. calling an external API, triggering webhooks to arbitrary customer endpoints).

In common parlance, my node generating the event would be the "client", and the remote (which I may or may not manage) would be the "server".

Parallel to this, it may be worth mentioning that for other reasons, we're starting to discuss doing classification of the IPs (local, private, public, multicast), which may help figure out which side is which.

[webmat]

Another note about this, more applicable for security, but also when monitoring network gear.

Who's the client and who's the server, if the flow event comes from an agent that's sitting in between? :-)

[robcowart]

Basically a server provides a service (a port or group of ports). Clients connect to those services. A server will only respond to a client. It doesn't initiate conversations. Conversely a client only listens for responses. It doesn't listen for arbitrary connection requests.

The determination of client and server can be quite tricky if you don't have a record of the initial packet transmitted (such as the SYN packet sent to initiate the TCP handshake). 20 years ago you could be >90% accurate simply by assuming the lower port value is the server and the larger value is the client. However with so many applications now listening on higher ports (e.g. ES 9200, LS 9600, Kafka 9092, etc.) you will get at best about 65% accuracy with with this method. Many log sources are bit more authoritative in this regard than flow records.

Basically there isn't a single method that works. A combination of data source specific methods that arrive at a consensus is usually necessary. With the solution we provide to our paying customers, we find that we are about 95% accurate out-of-the-box. With some tuning (it can be customized) 98-99% is possible.

@webmat can you provide a more specific example of what you are referring to regarding an "agent"?

[robcowart]

I will also add that local/private/public isn't much help when determining client/server. However reserved multicast and broadcast IP and MAC addresses will always be associated with the server end of the conversation. This is one input for the "consensus" method we use.

[robcowart]

The last point I will make is that it is not an either/or situation. While client/server is the preferred perspective for most use-cases, src/dst is needed for some types of threat detection.

Consider a few security related analytics scenarios...

• A port scan will be from client to server.
• However an amplification attack will look at sources to destinations, where the source port would be from a well known UDP service (e.g. 53 for DNS).

So depending on what we are looking for our analytics configuration will sometimes use src/dst and sometimes use client/server.

[ave19]

@webmat

in cases where the infrastructure under management is calling to the outside (e.g. calling an external API, triggering webhooks to arbitrary customer endpoints).

The host running the API service is also generating logs. From that service's perspective, it's the server (running a service) and your caller is the client.

In the events coming from your server, its the server and things that connect to it are clients.

[ave19]

@webmat

Who's the client and who's the server, if the flow event comes from an agent that's sitting in between? :-)

Whoever got the first SYN is the server. Generally, the lower port.

[edit: our pcap drop rates are really low, but not zero, so we might miss that SYN. See @robcowart comment. also also, with UDP you don't even get that. For a UDP service, unless you do protocol inspection, you can't really know whether the packet you saw was the request or the answer.]

The agent in the middle may not be able to tell, though. When we map source and destination to client and server, we don't delete the source and destination bits, those are the ones we're sure about!

[ave19]

@robcowart

The determination of client and server can be quite tricky

You are so right!

[ave19]

Honestly, I don't expect a flow's interpretation of client and server to be 100% accurate for all the reasons @robcowart points out.

Most of the time we're going to use those tags, we're applying them to logs coming from things like web servers. From inside a web server's event feed, the source and destination don't really apply. And, if you're one of ten servers running on a host, you might have a different server.ip from the others, and each of those different from host.ip on which you run, and agent.ip or device.ip where your logs get sent, or what have you.

It just makes a little space.

[ave19]

And for cyber reasons, having all of your servers (on whatever boxes) call all of their clients client allows us to more easily track one or more IPs that might be up to something by correlating logs from all the services running on all the hosts.

[robcowart]

I agree with you @ave19, for some data sources the client and server are clear. We still set source and destination fields, but will also set something like "[metadata][isServer]" => "destination". When the event hits the client/server determination logic, this flag will cause the more complicated logic to be bypassed, and the client/server fields to be set with a simple assignment.

[ave19]

@robcowart interesting... we have lots of different kinds of feeds, so lots of different parsing logic. most of the time, we can go straight in to the server.ip form.

[willemdh]

Although I can definitely understand your points @ave19, for me source and destination are more clear and less Confusing / ambigous then client and server. When 2 applications are exchanging data through an esb, id really prefer to be able to use source and destination objects in the esb logs. But then again I'm a system engineer, not a network engineer.

[ave19]

@willemdh I hear you, but think about UDP. Or think about DNS in particular. One system sends a request to another, and the other answers it. Do you swap source and destination on both sides of that? UDP is stateless, so you'd almost have to. That means the DNS server is in half the events on both sides. How do you pie chart where your requests are coming from in that scenario? Filter it in post? It is Elasticsearch I guess. It's not that doing source and destination is wrong in this scenario, it's that it's less easy to work with the data collected.

[willemdh]

@ave19 Ok, I can defintely use the client / servers approach for F5 / Palo Alto use cases.
In case I would need a 'non-connection' related source destination info,
I can always create my own (private) source and destination objects.

So, looking at #51 this is were ECS would go then

Field	Description	Type
connection.server.host.ip	IP address of the server.Can be one or multiple IPv4 or IPv6 addresses.	ip
connection.server.host.name	Hostname of the server.	keyword
connection.server.host.port	Port of the server.	long
connection.server.host.mac	MAC address of the server.	keyword
connection.server.host.domain	server domain.	keyword
connection.server.host.subdomain	server subdomain.	keyword
connection.client.host.ip	IP address of the client.Can be one or multiple IPv4 or IPv6 addresses.	ip
connection.client.host.name	Hostname of the client.	keyword
connection.client.host.port	Port of the client.	long
connection.client.host.mac	MAC address of the client.	keyword
connection.client.host.domain	client domain.	keyword
connection.client.host.subdomain	client subdomain.	keyword
connection.direction	Direction of the network traffic. Recommended values are:* inbound* outbound* unknown	keyword
connection.forwarded_ip	Host IP address when the client IP address is the proxy.	ip

Shouldn't we move network.session_id to the connection object too then? See #37