Support tracing live traffic in Traceflow

[jianjuns]

Describe what you are trying to solve
Support tracing real traffic going on between Pods, besides tracing the injected Traceflow packet.
Collect the captured packet headers for live-traffic tracing.
Support tracing only the packets dropped by NetworkPolicies.

Examples:

• I like to trace the TCP packet from Pod1 to Service1.
• I like to trace the first UDP packet from Pod1 in 10 minutes, and check the source/destination ports and packet length.
• I like to capture the first packet dropped by NetworkPolicies from Pod1 within 1 hour.

Describe the solution you have in mind
Add a flag - LiveTraffic - in Traceflow.Spec to indicate live-traffic tracing. When it is set, instead of generating and injecting a Traceflow packet, the sender Node should just add flows to tag the first packet of the first connection from the Source Pod that matches the Traceflow spec. The tag should be removed before the packet leaving the Antrea pipeline (e.g. outputing to the destination Pod, or sending to external).

Addd a new field to Traceflow.Status for collecting the (IP/TCP/UDP) header information of the captured packet. Antrea Agent on the Sender Node should report the captured packet headers to Traceflow.Status.CapturePacket.

Add a flag - DroppedOnly - Traceflow.Spec to indicate only the dropped packet should be traced/captured. Antrea Agent should add flows to capture only the packets dropped by NetworkPolicies (and match the Traceflow spec).

Add a Timeout attribute to Traceflow.Spec to specify the timeout time of a Traceflow session.

Extend antctl and Octant UI to support live-traffic Traceflow.

Example antctl command

# Capture the first dropped TCP packet from Pod client to Service web-server within 10m.
$ antctl traceflow -S client -D web-server --live-traffic -f tcp -t 10m --dropped-only
name: client-web-server-to-any-5sb65mnq
phase: Succeeded
source: client
destination: web-server
results:
- node: k8s2
  timestamp: 1617591772
  observations:
  - component: Forwarding
    componentInfo: Classification
    action: Received
  - component: NetworkPolicy
    componentInfo: IngressDefaultRule
    action: Dropped
capturedPacket:
  srcIP: 172.100.0.16
  dstIP: 172.100.1.13
  length: 60
  ipHeader:
    protocol: 6
    ttl: 62
    flags: 2
  tranportHeader:
    tcp:
      srcport: 47950
      dstport: 80
      flags: 2

[jianjuns]

• Support live-traffic Traceflow (Support Traceflow of live traffic #2005)
• Support configurable Traceflow timeout (TCP rejection can't work on Kind when the traffic mode is noEncap #2025)
• Enhance antctl traceflow command to support live-traffic Traceflow (TCP rejection can't work on Kind when the traffic mode is noEncap #2025)
• Report captured packet information and support capturing only dropped packet (Report captured packet in live-traffic Traceflow and support capturing only dropped packet #2029)
• Support live-traffic Tracflow for specific traffic from any Source or a source IP to a particular destination Pod (Support any source or IP address source in live-traffic Traceflow #2068)
• Extend Octant plugin to support live-traffic Traceflow

[srikartati]

Addd a new field to Traceflow.Status for collecting the (IP/TCP/UDP) header information of the captured packet. Antrea Agent on the Sender Node should report the captured packet headers to Traceflow.Status.CapturePacket.

Hi @jianjuns , Interesting feature with traceflow. Is it possible to capture this packet on the receiver node as well? I think this can be useful to figure out if the packet of a flow reached the destination or dropped somewhere in the underlay.

[jianjuns]

Yes, it does trace the forwarding path from sender to receiver just like a normal Traceflow, but I meant only the sender will report the packet header info.

[srikartati]

Thanks Jianjun. What I meant is could the antrea agent on the receiver node also report these packet headers with a different traceflow request?
If I am not wrong, you seem to be adding that functionality in this PR: #2068

[jianjuns]

So of now a sender/source Pod must be specified for a Traceflow, and then I feel it is good enough for only the sender to report packet headers (but the receiver will report NodeResult too which proves the packet is received on the receiver Node and is delivered to the destination Pod or dropped).
#2068 is to support capture a packet from any sender to a receiver, in which case only the receiver will report the packet.
Do the above answer your question?

[srikartati]

Got it. Thanks for clarifying, Jianjun.

[github-actions]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment, or this will be closed in 180 days