From 2a391818bd5a009371a1ed97c8e5d108cd6a6f83 Mon Sep 17 00:00:00 2001 From: Srikar Tati Date: Mon, 21 Sep 2020 17:23:10 -0700 Subject: [PATCH] Update flow exporter documentation Update the figure in flow exporter documentation --- docs/assets/flow_exporter.svg | 1278 +++++++++++++++++++++++++++++++ docs/network-flow-visibility.md | 60 +- 2 files changed, 1323 insertions(+), 15 deletions(-) create mode 100644 docs/assets/flow_exporter.svg diff --git a/docs/assets/flow_exporter.svg b/docs/assets/flow_exporter.svg new file mode 100644 index 00000000000..b8f90f1259d --- /dev/null +++ b/docs/assets/flow_exporter.svg @@ -0,0 +1,1278 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + K8s Cluster + Antrea Controller + + veth + veth + veth + OVS Bridge + Conntrack + Pod + AntreaDaemon Set + NODE + Antrea Agent + Antrea OVS + Kubelet + AntreaCNI + FlowExporter + K8sInfo + Kube-Proxy + IP Tables + OVS Daemons + IPFIX Flow CollectorAnd Flow Visualization(ELK Flow Collector) + IPFIX FlowRecords + WorkloadPods + PrometheusServer + ConnectionsMetrics + + diff --git a/docs/network-flow-visibility.md b/docs/network-flow-visibility.md index 0837a7f81bf..308b0946152 100644 --- a/docs/network-flow-visibility.md +++ b/docs/network-flow-visibility.md @@ -26,7 +26,7 @@ to enhance the performance and security aspects of Pod workloads. For visualizing the network flows, Antrea monitors the flows in Linux conntrack module. These flows are converted to flow records and are sent to the configured flow controller. High-level design is given below: -Network Flow Visibilty +![Flow Exporter Design](assets/flow_exporter.svg) ## Flow Exporter feature @@ -70,6 +70,50 @@ ConfigMap as shown below. We provide some examples for the parameter values in t Please note that the default values for `flowPollInterval` and `flowExportFrequency` parameters are set to 5s and 12, respectively. `flowCollectorAddr` is a required parameter that is necessary for the Flow Exporter feature to work. +### IPFIX Fields in a Flow Record + +There are 26 IPFIX fields in each flow record. They are from the standard IANA + registry, the reverse IANA registry and the Antrea registry for Antrea specific + fields. The reverse IANA registry fields are used for flows in the reverse + direction. +#### IANA Registry Fields + +| IPFIX Information Element| Enterprise ID | Field ID | Type | +|--------------------------|---------------|----------|----------------| +| flowStartSeconds | 0 | 150 | dateTimeSeconds| +| flowEndSeconds | 0 | 151 | dateTimeSeconds| +| sourceIPv4Address | 0 | 8 | ipv4Address | +| destinationIPv4Address | 0 | 12 | ipv4Address | +| sourceTransportPort | 0 | 7 | unsigned16 | +| destinationTransportPort | 0 | 11 | unsigned16 | +| protocolIdentifier | 0 | 4 | unsigned8 | +| packetTotalCount | 0 | 86 | unsigned64 | +| octetTotalCount | 0 | 85 | unsigned64 | +| packetDeltaCount | 0 | 2 | unsigned64 | +| octetDeltaCount | 0 | 1 | unsigned64 | + +#### Reverse IANA Registry Fields + +| IPFIX Information Element| Enterprise ID | Field ID | Type | +|--------------------------|---------------|----------|----------------| +| packetTotalCount | 29305 | 86 | unsigned64 | +| octetTotalCount | 29305 | 85 | unsigned64 | +| packetDeltaCount | 29305 | 2 | unsigned64 | +| octetDeltaCount | 29305 | 1 | unsigned64 | + +#### Antrea Registry Fields + +| IPFIX Information Element | Enterprise ID | Field ID | Type | +|---------------------------|---------------|----------|-------------| +| sourcePodNamespace | 55829 | 100 | string | +| sourcePodName | 55829 | 101 | string | +| destinationPodNamespace | 55829 | 102 | string | +| destinationPodName | 55829 | 103 | string | +| sourceNodeName | 55829 | 104 | string | +| destinationNodeName | 55829 | 105 | string | +| destinationClusterIP | 55829 | 106 | ipv4Address | +| destinationServicePortName| 55829 | 108 | string | + ### Supported capabilities Currently, the Flow Exporter feature provides visibility for Pod-to-Pod, Pod-to-Node, Node-to-Pod, Node-to-Node and Pod-to-Service @@ -105,20 +149,6 @@ supports Netflow v5/v9/v10(IPFIX) protocols for flow data collection. The flow exporter feature in Antrea Agent uses the IPFIX (Netflow v10) protocol to export flow records. -Exported IPFIX flow records contain the following Antrea-specific fields along -with standard IANA fields. - -| IPFIX Information Element | Enterprise ID | Field ID | Type | -|---------------------------|---------------|----------|-------------| -| sourcePodNamespace | 55829 | 100 | string | -| sourcePodName | 55829 | 101 | string | -| destinationPodNamespace | 55829 | 102 | string | -| destinationPodName | 55829 | 103 | string | -| sourceNodeName | 55829 | 104 | string | -| destinationNodeName | 55829 | 105 | string | -| destinationClusterIP | 55829 | 106 | ipv4Address | -| destinationServicePortName| 55829 | 108 | string | - [Elasticsearch](https://www.elastic.co/elasticsearch/), as a RESTful search engine, supports storing, searching and indexing records received. [Kibana](https://www.elastic.co/kibana/) is mainly for data visualization and