Network Topology Dashboard Update #488

Dhruv-J · 2023-09-27T08:33:40Z

This PR updates the Network Topology dashboard to include visualizations for
multiple networking layers. Specifically, the following has been changed:

The source code of DependencyPanel has been modified to include the
DiagramController class, which acts as a rendering controller for the
mermaid diagram. This is needed after an update to the mermaid library,
which changes the functionality of the existing mermaidAPI render function.
The @grafana-toolkit tool will be deprecated soon, and will be replaced
by @grafana/create-plugin. With the replacement, the directory structure of
the plugin has been significantly changed, with a .config folder containing
the specific configuration files for the plugin's build and run, with general
configuration files in the base directory used to extend the default configs.
A unit test for the DependencyPanel has been added, and it generates a
simple graph and checks the DOM for the expected graph string being present.

Additionally, this PR builds upon #417, which adds additional fields for L7
visibility.

codecov · 2023-09-27T16:36:20Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 70.60%. Comparing base (79010a7) to head (ea9b703).
Report is 24 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #488      +/-   ##
==========================================
+ Coverage   70.51%   70.60%   +0.09%     
==========================================
  Files          40       45       +5     
  Lines        5253     5511     +258     
  Branches        0       41      +41     
==========================================
+ Hits         3704     3891     +187     
- Misses       1372     1436      +64     
- Partials      177      184       +7

Flag	Coverage Δ		*Carryforward flag
kind-e2e-tests	`62.48% <ø> (ø)`		Carriedforward from 3dd2dc7
kind-multi-cluster-e2e-tests	`43.17% <ø> (ø)`
python-coverage	`55.75% <ø> (ø)`		Carriedforward from 3dd2dc7
unit-tests	`70.17% <ø> (+0.02%)`	⬆️

*This pull request uses carry forward flags. Click here to find out more.

see 5 files with indirect coverage changes

Dhruv-J · 2023-09-27T16:51:45Z

Currently, the legend is shown as a paragraph, but I can make it more visual as well. Also looking into making the mermaid charts themselves bigger

Dhruv-J · 2023-09-27T22:25:43Z

For reviewers, a lot of files needed to be added as a result of the yarn tool being updated from toolkit to the Grafana command itself. This update comes with a lot more surrounding configuration, and is necessary for anyone wanting to build the plugin locally. The contents of the .config/ folder can be ignored, as they were automatically generated by the command yarn create @grafana/plugin migrate. The additional configuration I needed to get it working was the file webpack.config.ts file. To sum up, the code outside of the plugin directory, and both DiagramController.tsx and DependencyPanel.tsx are the main files within the PR that need to be reviewed.

yuntanghsu · 2023-09-27T22:54:35Z

Why do we have duplicated connections in layer 4/7 Topology? Or are they two different connections?
For Layer 4 Topology, is svc_antrea-test-3 the endpoint of web-server-3?

Dhruv-J · 2023-09-27T22:57:56Z

Why do we have duplicated connections in layer 4/7 Topology? Or are they two different connections?

They are two different connections

For Layer 4 Topology, is svc_antrea-test-3 the endpoint of web-server-3?

Yes, the goal of the layer 4 topology is to show destination Pod and Service, I'll update the legend to reflect that.

heanlan · 2023-09-27T22:58:12Z

Please correct me if I'm wrong. I remember last time we decided to use the chord diagrams to show the flows and use a table to display all the L7 fields. Did I miss anything?

Dhruv-J · 2023-09-27T22:59:51Z

Please correct me if I'm wrong. I remember last time we decided to use the chord diagrams to show the flows and use a table to display all the L7 fields. Did I miss anything?

Not wrong at all, I'm splitting them up into two PRs, with the latter having the second dashboard with the chord diagram and httpvals table. Since httpvals contains nested JSON, some massaging to the data is required.

yuntanghsu

Some comments:

Do we need a .config folder? I don't see this folder under other plugins?
Do you think we can add these changes to the e2e test or UT?
Can you update network-flow-visibility.md to reflect these changes?

build/charts/theia/README.md

yuntanghsu · 2023-09-28T20:16:31Z

build/charts/theia/provisioning/dashboards/network_topology_dashboard.json

+            }
+          },
+          "queryType": "sql",
+          "rawSql": "SELECT sourcePodName, sourcePodLabels, sourcePodNamespace, sourceNodeName, destinationPodName, destinationPodLabels, destinationNodeName, destinationServicePortName, octetDeltaCount FROM flows\nWHERE sourcePodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND destinationPodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND ( destinationPodName != '' OR sourcePodName != '' )\nAND octetDeltaCount != 0\nAND httpVals == ''\nAND $__timeFilter(flowEndSeconds)\nORDER BY flowEndSeconds DESC",


Please correct me if I'm wrong.
I think we don't have case where destinationPodName == '' and sourcePodName == ''.?

I had it as a failsafe, but I'll remove that conditional

yuntanghsu · 2023-09-28T20:18:43Z

build/charts/theia/provisioning/dashboards/network_topology_dashboard.json

+        "type": "grafana-clickhouse-datasource",
+        "uid": "PDEE91DDB90597936"
+      },
+      "description": "The different colors of the lines mean different codes, but are labelled by the content length of the flow.\nGreen: Success",


Missing description for other colors?

I'll remove description, as the legend describes the colors anyways.

yuntanghsu · 2023-09-28T20:20:04Z

build/charts/theia/provisioning/dashboards/network_topology_dashboard.json

  "weekStart": ""
-}
+}


Missing empty line at the end of the file.

It's not fixed?

yuntanghsu · 2023-09-28T20:21:20Z

build/charts/theia/provisioning/datasources/migrators/000005_0-6-0.down.sql

@@ -212,3 +212,9 @@ ALTER TABLE tadetector_local
    DROP COLUMN aggType,
    DROP COLUMN direction,
    DROP COLUMN podName;
+ALTER TABLE flows


Should these changes be in 000005_0-6-0.down.sql or in next version like 000006_0-7-0.down.sql?

I think these are changes part of Tushar's commit and PR, I just have them in this PR as well.

PR has been updated, Please use the new fields and files

yuntanghsu · 2023-09-28T20:21:27Z

build/charts/theia/provisioning/datasources/migrators/000005_0-6-0.up.sql

@@ -151,3 +151,9 @@ ALTER TABLE tadetector_local
    ADD COLUMN aggType String,
    ADD COLUMN direction String,
    ADD COLUMN podName String;
+ALTER TABLE flows


yuntanghsu · 2023-09-28T20:21:35Z

build/charts/theia/values.yaml

@@ -211,7 +211,7 @@ grafana:
  installPlugins:
    - https://downloads.antrea.io/artifacts/grafana-custom-plugins/theia-grafana-sankey-plugin-1.0.2.zip;theia-grafana-sankey-plugin
    - https://downloads.antrea.io/artifacts/grafana-custom-plugins/theia-grafana-chord-plugin-1.0.1.zip;theia-grafana-chord-plugin
-    - https://downloads.antrea.io/artifacts/grafana-custom-plugins/theia-grafana-dependency-plugin-1.0.2.zip;theia-grafana-dependency-plugin
+    - https://github.com/Dhruv-J/grafana-dependency-plugin/archive/refs/tags/v0.zip;theia-grafana-dependency-plugin


yuntanghsu · 2023-09-28T20:23:26Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/.config/Dockerfile

+
+# Inject livereload script into grafana index.html
+USER root
+RUN sed -i 's/<\/body><\/html>/<script src=\"http:\/\/localhost:35729\/livereload.js\"><\/script><\/body><\/html>/g' /usr/share/grafana/public/views/index.html


Missing empty line here.

yuntanghsu · 2023-09-28T20:24:45Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/.eslintrc

@@ -0,0 +1,3 @@
+{
+  "extends": "./.config/.eslintrc"
+}


missing empty line not solved?

yuntanghsu · 2023-09-28T20:24:54Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/.prettierrc.js

-};
+  // Prettier configuration provided by Grafana scaffolding
+  ...require("./.config/.prettierrc.js")
+};


Dhruv-J · 2023-09-28T20:45:38Z

Do we need a .config folder? I don't see this folder under other plugins?

Yes, as I mentioned in my comment earlier, the .config folder is created as a result of upgrading the tool used to create, build, and sign plugins. The old tool will be deprecated soon, and there will be another PR that updates all the plugins to have such a configuration. Older plugins don't have this since they haven't been rebuilt or upgraded to have it, but it is a necessary folder for anyone wishing to build the plugin locally or make changes to its source code. The additional config file I added was weback.config.ts within the base grafana-dependency-plugin directory, which was provided by a Grafana dev.

Do you think we can add these changes to the e2e test or UT?

The network topology dashboard already has a unit test, which I included when I originally made the dashboard. It simply checks the query to see if it returns expected values.

Can you update network-flow-visibility.md to reflect these changes?

Yes, will do.

tushartathgur · 2023-10-02T17:18:22Z

build/yamls/flow-visibility.yml

@@ -3571,11 +3626,11 @@ data:
                }
              },
              "queryType": "sql",
-              "rawSql": "SELECT sourcePodName, sourcePodLabels, sourcePodNamespace, sourceNodeName, destinationPodName, destinationPodLabels, destinationNodeName, destinationServicePortName, octetDeltaCount FROM flows\nWHERE sourcePodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND destinationPodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND destinationPodName != ''\nAND sourcePodName != ''\nAND octetDeltaCount != 0\nAND $__timeFilter(flowEndSeconds)\nORDER BY flowEndSeconds DESC",
+              "rawSql": "SELECT sourcePodName, sourcePodLabels, sourcePodNamespace, sourceNodeName, sourceIP, sourceTransportPort, destinationPodName, destinationPodLabels, destinationNodeName, destinationServicePortName, destinationIP, destinationTransportPort, octetDeltaCount, httpVals FROM flows\nWHERE sourcePodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND destinationPodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND ( destinationPodName != '' OR sourcePodName != '' )\nAND octetDeltaCount != 0\nAND httpVals != ''\nAND $__timeFilter(flowEndSeconds)\nORDER BY flowEndSeconds DESC",


Not getting l7ProtocolName? how are we checking if the flow is L7 and which protocol is used in the flow?

Since only http protocols are being exported, right now I only check for httpVals existing. Once new protocols are added, their JSON will need to be parsed differently as well, so a patch PR can be made.

tushartathgur · 2023-10-02T17:20:32Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/.config/.prettierrc.js

+  "singleQuote": true,
+  "useTabs": false,
+  "tabWidth": 2
+};


Resolve all empty line issues

tushartathgur · 2023-10-02T17:24:21Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/src/DependencyPanel.tsx

+        } else {
+          graphLine = graphLine + destinationIP + ':' + destinationPort;
+        }
+        // graphLine = sourceIP + ':' + sourcePort + ' --' + frameHttpInfo.ContentLength + '--> ' + destinationIP + ':' + destinationPort + '\n';


remove this line?

tushartathgur · 2023-10-02T17:28:10Z

build/yamls/flow-visibility.yml

+                }
+              },
+              "queryType": "sql",
+              "rawSql": "SELECT sourcePodName, sourcePodLabels, sourcePodNamespace, sourceNodeName, destinationPodName, destinationPodLabels, destinationNodeName, destinationServicePortName, octetDeltaCount FROM flows\nWHERE sourcePodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND destinationPodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND ( destinationPodName != '' OR sourcePodName != '' )\nAND octetDeltaCount != 0\nAND httpVals == ''\nAND $__timeFilter(flowEndSeconds)\nORDER BY flowEndSeconds DESC",


Use l7ProtocolName to filter L4 flows

tushartathgur · 2023-10-02T17:35:15Z

build/charts/theia/provisioning/datasources/migrators/000005_0-6-0.down.sql

@@ -212,3 +212,9 @@ ALTER TABLE tadetector_local
    DROP COLUMN aggType,
    DROP COLUMN direction,
    DROP COLUMN podName;
+ALTER TABLE flows


PR has been updated, Please use the new fields and files

tushartathgur · 2023-10-02T17:35:50Z

build/charts/theia/provisioning/datasources/create_table.sh

@@ -80,6 +80,8 @@ clickhouse client -n -h 127.0.0.1 <<-EOSQL
        clusterUUID String,
        egressName String,
        egressIP String,
+        isL7 Bool,


l7ProtocolName?

tushartathgur · 2023-10-02T17:36:51Z

build/charts/theia/provisioning/dashboards/network_topology_dashboard.json

@@ -60,11 +101,11 @@
            }
          },
          "queryType": "sql",
-          "rawSql": "SELECT sourcePodName, sourcePodLabels, sourcePodNamespace, sourceNodeName, destinationPodName, destinationPodLabels, destinationNodeName, destinationServicePortName, octetDeltaCount FROM flows\nWHERE sourcePodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND destinationPodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND destinationPodName != ''\nAND sourcePodName != ''\nAND octetDeltaCount != 0\nAND $__timeFilter(flowEndSeconds)\nORDER BY flowEndSeconds DESC",
+          "rawSql": "SELECT sourcePodName, sourcePodLabels, sourcePodNamespace, sourceNodeName, sourceIP, sourceTransportPort, destinationPodName, destinationPodLabels, destinationNodeName, destinationServicePortName, destinationIP, destinationTransportPort, octetDeltaCount, httpVals FROM flows\nWHERE sourcePodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND destinationPodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND ( destinationPodName != '' OR sourcePodName != '' )\nAND octetDeltaCount != 0\nAND httpVals != ''\nAND $__timeFilter(flowEndSeconds)\nORDER BY flowEndSeconds DESC",


Use l7ProtocolName
Refer to Adding L7 visibility fields

I'll need to check httpVals regardless, since I need to parse its JSON in a specific manner, so I assumed it was redundant to check both. If needed, I can make a later patch that checks the protocolName and parses the JSON accordingly.

tushartathgur · 2023-10-02T17:38:41Z

build/charts/theia/provisioning/dashboards/network_topology_dashboard.json

+            }
+          },
+          "queryType": "sql",
+          "rawSql": "SELECT sourcePodName, sourcePodLabels, sourcePodNamespace, sourceNodeName, destinationPodName, destinationPodLabels, destinationNodeName, destinationServicePortName, octetDeltaCount FROM flows\nWHERE sourcePodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND destinationPodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND ( destinationPodName != '' OR sourcePodName != '' )\nAND octetDeltaCount != 0\nAND httpVals == ''\nAND $__timeFilter(flowEndSeconds)\nORDER BY flowEndSeconds DESC",


Dhruv-J · 2023-10-04T22:27:22Z

Will further update README and changelog to reflect latest changes.

yuntanghsu · 2023-10-10T17:52:28Z

ci/kind/test-multi-cluster.sh

@@ -33,7 +33,7 @@ function print_usage {


 TESTBED_CMD=$(dirname $0)"/kind-setup.sh"
-FLOW_VISIBILITY_CH_ONLY_CMD=$(dirname $0)"/../../hack/generate-manifest.sh --no-grafana --node-port --secure-connection"
+FLOW_VISIBILITY_CH_ONLY_CMD=$(dirname $0)"/../../hack/generate-manifest.sh --ch-only --node-port --secure-connection"


Remove changes not belong to this PR? Do rebase?

yuntanghsu · 2023-10-10T17:58:44Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/README.md

@@ -114,7 +143,8 @@ installed panels will appear. For more information, visit the docs on [Grafana p

 Users can customize the panel by editing its options and choosing to group the
 diagram based on a chosen Pod label. It is also possible to change the color
-of the Pod squares in the diagram.
+of the Pod squares in the diagram. Additionally, users can switch betwen layer
+4 and layer 7 visualizations.

 <img src="https://user-images.githubusercontent.com/10016630/233191376-7cf471b8-5e3e-473a-8696-da25ab981066.png" width="400" alt="Panel Option Editor View">



I think we need to update the section 4 as you already migrate the dependency-plugin from grafana/toolkit to create-plugin?

yuntanghsu · 2023-10-10T22:07:32Z

.github/workflows/kind.yml

@@ -217,6 +217,10 @@ jobs:
          docker tag antrea/theia-clickhouse-monitor:latest projects.registry.vmware.com/antrea/theia-clickhouse-monitor:latest
          docker load -i clickhouse-server.tar
          docker tag antrea/theia-clickhouse-server:latest projects.registry.vmware.com/antrea/theia-clickhouse-server:latest
+      - name: Remove Theia images tar files


I think we need to update the typescript.yaml in workflow as well?

Dhruv-J · 2023-10-18T17:32:48Z

Unit tests are proving to be challenging to setup with updated toolkit versions, will make patch PR for them separately.

yuntanghsu · 2023-10-19T06:22:48Z

build/charts/theia/provisioning/datasources/migrators/000005_0-6-0.down.sql

Do you know why we have these two empty files here?

not solved?

heanlan

Could you add more details into the PR descriptions? including the your changes from the following aspects and why we need those changes:

data schema
dependency plugin source code changes, with newly added diagramController
dependency upgrade of dependency plugin, which introduces .config, jest folders, etc
newly added UT

heanlan · 2023-10-25T17:22:34Z

build/charts/theia/provisioning/datasources/migrators/000005_0-6-0.down.sql

not solved?

.github/workflows/typescript.yml

heanlan · 2023-11-01T21:35:31Z

build/charts/theia/provisioning/dashboards/network_topology_dashboard.json

+            }
+          },
+          "queryType": "sql",
+          "rawSql": "SELECT sourcePodName, sourcePodLabels, sourcePodNamespace, sourceNodeName, destinationPodName, destinationPodLabels, destinationNodeName, destinationServicePortName, octetDeltaCount FROM flows\nWHERE sourcePodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND destinationPodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')\nAND ( destinationPodName != '' OR sourcePodName != '' )\nAND octetDeltaCount != 0\nAND httpVals == ''\nAND $__timeFilter(flowEndSeconds)\nORDER BY flowEndSeconds DESC",


Maybe I missed something, but why we enforce ( destinationPodName != '' OR sourcePodName != '' ) for layer 4 but accept ip+port for layer 7?

I'll remove them for both, that condition should be true by default

heanlan · 2023-11-01T22:49:41Z

build/charts/theia/provisioning/datasources/migrators/000005_0-6-0.up.sql

empty file?

heanlan · 2023-11-01T22:56:00Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/.eslintrc

@@ -0,0 +1,3 @@
+{
+  "extends": "./.config/.eslintrc"
+}


missing empty line not solved?

heanlan · 2023-11-01T23:02:17Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/src/DependencyPanel.test.tsx

+    props.width = 600,
+    props.height = 600;
+    props.options = {};
+    let component = shallow(<DependencyPanel {...props} />);


@yuntanghsu could you help take a look at this UT as you recently modified chord/sankey plugins UT and have some experience with how we should design react UTs? I feel like we can also reference to your latest change https://github.com/antrea-io/theia/blob/main/plugins/grafana-custom-plugins/grafana-chord-plugin/src/ChordPanel.test.tsx, but not very sure whether it's also applicable here

I think we are doing a similar test, which verifies the input data reflected in the output component.
But I think we can also verify the context for [boxColor, layerLevelFour, themeColors]? And maybe we can add one more input for layerSeven as the format of GraphString of these two layer graphs are different?

heanlan · 2023-11-01T23:04:35Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/src/DiagramController.tsx

+
+  renderCallback(svgCode: string, bindFunctions: any) {
+    if (this && bindFunctions) {
+      console.log('binding diagram functions');


debug log can be removed?

heanlan · 2023-11-01T23:09:19Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/src/DependencyPanel.tsx

  let boxColor;
+  let layerLevel;


Why we need an extra var layerLevel? Can we just reuse options.layerFour this boolean var?

heanlan · 2023-11-01T23:21:44Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/src/DependencyPanel.tsx

+        return sourcePodName;
+      }
+
+      let groupByPodLabel = options.groupByPodLabel;


same question, can we reuse options.groupByPodLabel without introducing new variable?

yes, i can change this one

heanlan · 2023-11-01T23:29:24Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/src/DependencyPanel.tsx

+  function layerSevenGraph() {
+    function getColorFromStatus(httpStatus: string) {
+      // colors that correspond to each of the types of http response code; i.e. 4xx and 5xx codes return red to symbolize errors
+      const colors = ['orange', 'green', 'blue', 'red', 'red'];


I'd suggest just use a map data structure or case switch statement for this function, rather than the status first char matching colors list index.

I was thinking of checking the first character because it would allow all response codes that are within the 100 possible codes range to be represented. Since all 4xx codes are error codes, and all 2xx are success codes, I just checked the first character. I can put the underlying logic into a case switch however, that would be cleaner.

Checking the first char is ok. But I think the list index thing is not a common practice.

I suppose we can combine the first char matching with a case switch statement.

heanlan · 2023-11-01T23:42:07Z

A general note: If the pull request isn't prepared for review yet, feel free to change its status to draft. If it is ready, I suggest consolidating certain intermediate and less relevant commits that might not provide meaningful information to reviewers.

This PR updates the Network Topology dashboard to include visualizations for multiple networking layers. It also updates the legends of both graphs to improve readability. Signed-off-by: Dhruv-J <dhruvj@vmware.com>

Signed-off-by: Dhruv-J <dhruvj@vmware.com>

yuntanghsu · 2023-11-09T19:52:15Z

build/charts/theia/provisioning/dashboards/network_topology_dashboard.json

  "weekStart": ""
-}
+}


It's not fixed?

yuntanghsu · 2023-11-09T19:55:49Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/.prettierrc.js

-};
+  // Prettier configuration provided by Grafana scaffolding
+  ...require("./.config/.prettierrc.js")
+};


yuntanghsu · 2023-11-09T19:58:24Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/README.md

-AND octetDeltaCount != 0
-AND $__timeFilter(flowEndSeconds)
+WHERE sourcePodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')
+   AND destinationPodNamespace NOT IN ('kube-system', 'flow-visibility', 'flow-aggregator')


For Kind cluster, there will be 1 more Namespace 'local-path-storage'. Do we want to exclude that as well?

yes, how can I test this change locally?

yuntanghsu · 2023-11-09T20:02:54Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/package.json

-    "start": "yarn watch"
+    "build": "webpack -c ./webpack.config.ts --env production",
+    "dev": "webpack -w -c ./webpack.config.ts --env development",
+    "e2e": "yarn exec cypress install && yarn exec grafana-e2e run",


Maybe you can follow this PR #587 to remove some unused scripts and dependencies in this PR to avoid some Vulnerability alerts bumped up after this PR is merged.
Note:
Not all dependencies detected by depcheck is unused.

yuntanghsu · 2023-11-09T20:10:49Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/src/DependencyPanel.test.tsx

+    props.width = 600,
+    props.height = 600;
+    props.options = {};
+    let component = shallow(<DependencyPanel {...props} />);


I think we are doing a similar test, which verifies the input data reflected in the output component.
But I think we can also verify the context for [boxColor, layerLevelFour, themeColors]? And maybe we can add one more input for layerSeven as the format of GraphString of these two layer graphs are different?

yuntanghsu · 2023-11-09T20:10:59Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/src/DependencyPanel.test.tsx

+      renderedHtmlString.includes(expectedGraphString)
+    ).toEqual(true);
+  });
+});


empty line.

yuntanghsu · 2023-11-09T20:24:02Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/tsconfig.json

-  }
-}
+  "extends": "./.config/tsconfig.json"
+}


yuntanghsu · 2023-11-09T20:24:09Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/webpack.config.ts

+  });
+};
+
+export default config;


Dhruv-J · 2023-11-14T16:16:22Z

But I think we can also verify the context for [boxColor, layerLevelFour, themeColors]?

Those are used directly into the theme, and in order to test them, the mermaid components would directly need to be tested, which doesn't seem possible because the component gets split up upon rendering in the DOM, and cannot be checked for directly.

And maybe we can add one more input for layerSeven as the format of GraphString of these two layer graphs are different?

Yes, will do

Signed-off-by: Dhruv-J <dhruvj@vmware.com>

heanlan · 2023-11-15T22:42:23Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/src/DependencyPanel.tsx

+      // colors that correspond to each of the types of http response code; i.e. 4xx and 5xx codes return red to symbolize errors
+      const codeToColor = {1:'orange', 2:'green', 3:'blue', 4:'red', 5:'red'};
+      let statusType = +httpStatus.charAt(0);
+      if (statusType < 1 || statusType > 5) {


Is it possible for us to get statusType < 1 or > 5? Is there such a http status code? I see you added it for indicating errors, but I wonder whether it is possible to happen, if it does happen, would it be better to catch this error in Flow Exporter or Flow Aggregator? cc @tushartathgur

I don't think it should, but we can double check with @tushartathgur.

heanlan · 2023-11-15T22:49:46Z

plugins/grafana-custom-plugins/grafana-dependency-plugin/src/DiagramController.tsx

+            ref={this.setDiagramRef}
+            className={`diagram-seven`}
+          ></div>
+          <div><p>In this graph, each flow between a source and destination is represented by an arrow labelled with the http content length of the data being sent.


Do we need a legend/sentence to explain how the colors are mapped to http codes, e.g. 1xx(informational response), 2xx(success), etc?

Yes, I can add a sentence below at the end.

Signed-off-by: Dhruv-J <dhruvj@vmware.com>

Dhruv-J · 2023-12-12T17:55:15Z

Instructions for those wanting to deploy the solution for themselves:

pull Antrea PR #5218
pull Theia PR Adding new Clickhouse fields for NP visibility #417 and this PR
execute the included script for build_antrea.sh.zip
deploy flow-visibility.yml from the new theia PR
apply the following: pods.yaml.zip
annotate the pods with the following command: kubectl pod annotate <pod-name> -n <namespace> [visibility.antrea.io/enable-l7=both](http://visibility.antrea.io/enable-l7=both)
kubectl exec into the applied admin pod and run the command curl example.com to get one row, or curl example.com example.com example.com to get multiple rows (it'll take about 2 minutes to populate clickhouse)
open the network topology dashboard to view http flows

heanlan

LGTM

Let's wait for all the required PRs getting merged first and give the final approval on this

github-actions · 2024-03-15T00:04:44Z

This PR is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

Dhruv-J force-pushed the dashboard-improvements branch from f63798e to 8fce175 Compare September 27, 2023 16:32

Dhruv-J requested review from heanlan and tushartathgur September 27, 2023 16:51

Dhruv-J requested a review from yuntanghsu September 28, 2023 18:10

yuntanghsu reviewed Sep 28, 2023

View reviewed changes

tushartathgur reviewed Oct 2, 2023

View reviewed changes

Dhruv-J force-pushed the dashboard-improvements branch from 2482c26 to d4526d3 Compare October 3, 2023 16:32

Dhruv-J requested a review from yuntanghsu October 10, 2023 16:58

yuntanghsu reviewed Oct 10, 2023

View reviewed changes

Dhruv-J force-pushed the dashboard-improvements branch from c910962 to 7a79147 Compare October 17, 2023 18:06

yuntanghsu reviewed Oct 19, 2023

View reviewed changes

heanlan reviewed Nov 1, 2023

View reviewed changes

Dhruv-J force-pushed the dashboard-improvements branch 2 times, most recently from 3df3912 to 72f071e Compare November 6, 2023 19:34

Network Topology Dashboard Update

0b94865

This PR updates the Network Topology dashboard to include visualizations for multiple networking layers. It also updates the legends of both graphs to improve readability. Signed-off-by: Dhruv-J <dhruvj@vmware.com>

Dhruv-J force-pushed the dashboard-improvements branch from 72f071e to 0b94865 Compare November 6, 2023 19:36

addressed feedback and updated unit tests

cafc006

Signed-off-by: Dhruv-J <dhruvj@vmware.com>

yuntanghsu reviewed Nov 9, 2023

View reviewed changes

Dhruv-J added 2 commits November 15, 2023 10:37

feedback fix

42983be

Signed-off-by: Dhruv-J <dhruvj@vmware.com>

flow-vis yml update

3dd2dc7

Signed-off-by: Dhruv-J <dhruvj@vmware.com>

heanlan reviewed Nov 15, 2023

View reviewed changes

http codes included in legend

ea9b703

Signed-off-by: Dhruv-J <dhruvj@vmware.com>

Dhruv-J mentioned this pull request Nov 21, 2023

Application Layer Dashboard #523

Closed

Dhruv-J requested review from heanlan and yuntanghsu December 11, 2023 18:29

heanlan reviewed Dec 15, 2023

View reviewed changes

github-actions bot added the lifecycle/stale label Mar 15, 2024

github-actions bot closed this Jun 13, 2024

Network Topology Dashboard Update #488

Network Topology Dashboard Update #488

Conversation

Dhruv-J commented Sep 27, 2023 • edited Loading

codecov bot commented Sep 27, 2023 • edited Loading

Codecov Report

Dhruv-J commented Sep 27, 2023 • edited Loading

Dhruv-J commented Sep 27, 2023 • edited Loading

yuntanghsu commented Sep 27, 2023

Dhruv-J commented Sep 27, 2023 • edited Loading

heanlan commented Sep 27, 2023

Dhruv-J commented Sep 27, 2023

yuntanghsu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tushartathgur Oct 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dhruv-J commented Sep 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tushartathgur Oct 2, 2023 • edited Loading

Choose a reason for hiding this comment

Dhruv-J commented Oct 4, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dhruv-J commented Oct 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heanlan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dhruv-J Nov 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heanlan commented Nov 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dhruv-J commented Nov 14, 2023

Dhruv-J commented Sep 27, 2023 •

edited

Loading

codecov bot commented Sep 27, 2023 •

edited

Loading

Dhruv-J commented Sep 27, 2023 •

edited

Loading

Dhruv-J commented Sep 27, 2023 •

edited

Loading

Dhruv-J commented Sep 27, 2023 •

edited

Loading

tushartathgur Oct 2, 2023 •

edited

Loading

Dhruv-J commented Sep 28, 2023 •

edited

Loading

tushartathgur Oct 2, 2023 •

edited

Loading

Dhruv-J Nov 7, 2023 •

edited

Loading

heanlan commented Nov 1, 2023 •

edited

Loading

Dhruv-J commented Dec 12, 2023 •

edited

Loading

heanlan left a comment •

edited

Loading