Packetbeat flows enhancements

[urso]

Flows support in 5.0 is quite rudimentary, not really taking any network layers (arbitrary timeout) into account and only counting packets and bytes. Plus, there is a parent-child relation between flows and transactions, not yet represented in packetbeat events being published.

List of flow proposed enhancements:

• pass active flow to packet analyzer:
  • add protocol specific stats to flows (e.g. number of transactions with success/failure/dropped due to packet loss/timed out)
  • add flow id to protocol transaction events to establish some relationship between flows and transaction
• add support to tie flow lifetime to connection status:
  • Flows for TCP/SCTP streams do not timeout while TCP connection is active
  • Protocols on top of UDP managing connection state should be able to disable flow timeout
  • Stop flow if connection is closed or connection attempt failed:
    • for TCP stop flow on RST or normal TCP shutdown
    • consider ICMP messages if TCP/UDP port is not reachable
    • timeout flow in case of handshake being incomplete and no data being send in either direction
  • add indicator (string?) to indicate the reason a flow has ended
  • add indicator if connection has been good (established TCP connection), in case it ended forcefully
  • add indicator for flow starting to capture an older TCP connection (not having seen the connection attempt itself, as packetbeat was started after)
• report TCP level stats:
  • count flags usage (SYN, RST, FIN, PUSH)
  • report bytes lost due to packet-loss
  • report packets fully/partially resend (number of packets and bytes)

[tbragin]

@urso @adriansr Does this pull also partially address this issue? #5476

[urso]

@tbragin No. Flows support and integrating flows into application protocols is about collecting metrics only.

If using Packetbeat and not interested in any of the supported protocols can we have a generic "TCP" option where we can specify a list of ports to be able to filter on specific traffic flows before the analysis stage.

[urso]

@london2016 this can already be done by configuring your custom packet filter in the device configs. Please checkout the forums if you need any help.

[q2dg]

Well...version 7.0 is here and nothing has been resolved about this issue yet...

[Umarhayat3]

PB version 7.3.1 has been released but this issue is still there and many are facing problems due to lack of information in debugging mode. This flow enhancement is necessary for troubleshooting.

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[faec]

Adding this to triage because we're still seeing requests for ways to measure tcp connection drops in particular and it looks like this fell off the radar for a while.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.