Add AWS VPC Flow Log integration (#698)

* Add AWS VPC Flow Log integration Signed-off-by: Haidong <whaidong@amazon.com> * init assets Signed-off-by: Haidong <whaidong@amazon.com> * add more visualizations Signed-off-by: Haidong <whaidong@amazon.com> * add readme for assets and aws vpc flow logs Signed-off-by: Haidong <whaidong@amazon.com> --------- Signed-off-by: Haidong <whaidong@amazon.com> Co-authored-by: Haidong <whaidong@amazon.com> (cherry picked from commit dd278ad)
opensearch-project · Jul 21, 2023 · 3009d8d · 3009d8d
1 parent 667b6fa
commit 3009d8d
Show file tree

Hide file tree

Showing 12 changed files with 2,155 additions and 0 deletions.
diff --git a/server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/README.md b/server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/README.md
@@ -0,0 +1,32 @@
+# AWS VPC Flow Logs Integration Assets
+
+API: http://osd:5601/api/saved_objects/_import?overwrite=true
+
+- [Assets](aws_vpc_flow-1.0.0.ndjson)
+
+## Asset List
+The next table details the assets
+
+| Name                          |     Type      |                                 Description                                 |
+|-------------------------------|:-------------:|:---------------------------------------------------------------------------:|
+| `ss4o_logs_vpc-aws_vpc_flow-*-*` | index-pattern |                              The Index Pattern                              |
+| `AWS VPC Flow Logs Overview`  |   dashboard   |               The pre-canned dashboard for AWS VPC flow logs                |
+| `[AWS VPC Flow Logs] Filters` | visualization |       [Controls] Interactive controls for easy dashboard manipulation       |
+| `[AWS VPC Flow Logs] Total Requests` | visualization |                   [Metric] Total requests through the VPC                   |
+| `[AWS VPC Flow Logs] Request History` | visualization |           [Vertical Bar] Number of Requests counted against time            |
+| `[AWS VPC Flow Logs] Requests by VPC ID` | visualization |              [Pie] Compare parts of Requests from each VPC ID               |
+| `[AWS VPC Flow Logs] Total Requests By Action` | visualization |                  [Metric] Number of Accept/Reject requests                  |
+| `[AWS VPC Flow Logs] Bytes` | visualization |              [Line] Trend of bytes transferred during the flow              |
+| `[AWS VPC Flow Logs] Packets` | visualization |             [Line] Trend of Packets transferred during the flow             |
+| `[AWS VPC Flow Logs] Bytes Metric` | visualization |       [Metric] Total ingress/egress bytes transferred during the flow       |
+| `[AWS VPC Flow Logs] Requests by Direction` | visualization |               [Pie] Compare parts of ingress/egress requests                |
+| `[AWS VPC Flow Logs] Requests by Direction Metric` | visualization |                 [Metric] Number of ingress/egress requests                  |
+| `[AWS VPC Flow Logs] Top Source Bytes` | visualization |   [Table] Top 10 source with number of bytes transferred during the flow    |
+| `[AWS VPC Flow Logs] Top Destination Bytes` | visualization | [Table] Top 10 destination with number of bytes transferred during the flow |
+| `[AWS VPC Flow Logs] Top Sources` | visualization |     [Table] Top 10 source with number of requests send during the flow      |
+| `[AWS VPC Flow Logs] Top Destinations` | visualization |   [Table] Top 10 destination with number of requests send during the flow   |
+| `[AWS VPC Flow Logs] Flow` | visualization |           [Vega] Illustrates the flow from Source to Destination            |
+| `[AWS VPC Flow Logs] Heat Map` | visualization |                [Heat Map] Heat Map of source and destination                |
+| `[AWS VPC Flow Logs] Top Source AWS Services` | visualization |              [Pie] Compare parts of AWS service as flow source              |
+| `[AWS VPC Flow Logs] Top Destination AWS Services` | visualization |           [Pie] Compare parts of AWS service as flow destination            |
+| `[AWS VPC Flow Logs] General Search` |    search     |                 The pre-canned search for AWS VPC flow logs                 |
diff --git a/...r/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/aws_vpc_flow-1.0.0.ndjson b/...r/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/aws_vpc_flow-1.0.0.ndjson
diff --git a/server/adaptors/integrations/__data__/repository/aws_vpc_flow/aws_vpc_flow-1.0.0.json b/server/adaptors/integrations/__data__/repository/aws_vpc_flow/aws_vpc_flow-1.0.0.json
@@ -0,0 +1,53 @@
+{
+  "name": "aws_vpc_flow",
+  "version": "1.0.0",
+  "displayName": "AWS VPC Flow",
+  "description": "AWS VPC Flow log collector",
+  "license": "Apache-2.0",
+  "type": "logs_vpc",
+  "author": "Haidong Wang",
+  "sourceUrl": "https://github.com/opensearch-project/observability/tree/2.x/integrations/aws_vpc_flow_log",
+  "statics": {
+    "logo": {
+      "annotation": "AWS VPC Logo",
+      "path": "logo.svg"
+    },
+    "gallery": [
+      {
+        "annotation": "AWS VPC Flow Log Dashboard",
+        "path": "dashboard1.png"
+      }
+    ]
+  },
+  "components": [
+    {
+      "name": "aws_vpc_flow",
+      "version": "1.0.0"
+    },
+    {
+      "name": "cloud",
+      "version": "1.0.0"
+    },
+    {
+      "name": "communication",
+      "version": "1.0.0"
+    },
+    {
+      "name": "logs_vpc",
+      "version": "1.0.0"
+    },
+    {
+      "name": "s3",
+      "version": "1.0.0"
+    }
+  ],
+  "assets": {
+    "savedObjects": {
+      "name": "aws_vpc_flow",
+      "version": "1.0.0"
+    }
+  },
+  "sampleData": {
+    "path": "sample.json"
+  }
+}