[Citrix ADC] Add Integration Package with lbvserver Data Stream (#4418)

* Add lbvserver data stream

* update changelog.yml and codeowners

* update config.yml file

* minor changes in README.md

* add and update dashboards

* Update visualization titles and screenshot

* minor changes in dashboards screenshot

* minor changes in README.md

* minor changes in README.md

Author: harnish-crest

packages/citrix_adc/_dev/build/docs/README.md

@@ -4,13 +4,13 @@
 
 The Citrix ADC integration allows you to monitor your Citrix ADC instance. Citrix ADC is an application delivery controller that performs application-specific traffic analysis to intelligently distribute, optimize, and secure Layer 4 - Layer 7 (L4–L7) network traffic for web applications.
 
-Use the Citrix ADC integration to collect metrics related to the interface and service. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference logs when troubleshooting an issue.
+Use the Citrix ADC integration to collect metrics related to the interface, lbvserver, and service. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference logs when troubleshooting an issue.
 
 ## Data streams
 
 The Citrix ADC integration collects metrics data.
 
-Metrics give you insight into the statistics of the Citrix ADC. Metrics data streams collected by the Citrix ADC integration include [interface](https://developer-docs.citrix.com/projects/netscaler-nitro-api/en/12.0/statistics/network/interface/) and [service](https://developer-docs.citrix.com/projects/netscaler-nitro-api/en/12.0/statistics/basic/service/), so that the user could monitor and troubleshoot the performance of the Citrix ADC instances.
+Metrics give you insight into the statistics of the Citrix ADC. Metrics data streams collected by the Citrix ADC integration include [interface](https://developer-docs.citrix.com/projects/netscaler-nitro-api/en/12.0/statistics/network/interface/), [lbvserver](https://developer-docs.citrix.com/projects/netscaler-nitro-api/en/12.0/statistics/lb/lbvserver/), and [service](https://developer-docs.citrix.com/projects/netscaler-nitro-api/en/12.0/statistics/basic/service/), so that the user could monitor and troubleshoot the performance of the Citrix ADC instances.
 
 Note:
 - Users can monitor and see the metrics inside the ingested documents for Citrix ADC in the logs-* index pattern from `Discover`.
@@ -33,15 +33,23 @@ Example Host Configuration: `http://localhost:9080`
 
 ### Interface
 
-This is the `interface` data stream. The Citrix ADC interfaces are numbered in slot/port notation. In addition to modifying the characteristics of individual interfaces, you can configure virtual LANs to restrict traffic to specific groups of hosts.
+This is the `interface` data stream. The Citrix ADC interfaces are numbered in slot/port notation. In addition to modifying the characteristics of individual interfaces, you can configure virtual LANs to restrict traffic to specific groups of hosts. `interface` data stream collects metrics related to id, state, inbound packets, outbound packets, and received packets.
 
 {{event "interface"}}
 
 {{fields "interface"}}
 
+### Load Balancing Virtual Server
+
+This is the `lbvserver` data stream. The load balancing server is logically located between the client and the server farm, and manages traffic flow to the servers in the server farm. `lbvserver` data stream collects metrics related to name, state, client connections, requests, and responses.
+
+{{event "lbvserver"}}
+
+{{fields "lbvserver"}}
+
 ### Service
 
-This is the `service` data stream. With the help of the service endpoint, metrics like throughput, client-server connections, request bytes can be collected along with other statistics for Service resources.
+This is the `service` data stream. With the help of the service endpoint, metrics like throughput, client-server connections, request bytes can be collected along with other statistics for Service resources. `service` data stream collects metrics related to name, IP address, port, throughput, and transactions.
 
 {{event "service"}}
 

packages/citrix_adc/_dev/deploy/docker/docker-compose.yml

@@ -11,3 +11,9 @@ rules:
       - status_code: 200
         body: |-
           {"errorcode": 0, "message": "Done", "severity": "NONE", "service": [{"name": "nshttpd-gui-127.0.0.1-80", "throughput": "0", "throughputrate": 0, "avgsvrttfb": "34", "primaryipaddress": "127.0.0.1", "primaryport": 80, "servicetype": "HTTP", "state": "UP", "totalrequests": "15133", "requestsrate": 0, "totalresponses": "15133", "responsesrate": 0, "totalrequestbytes": "8334520", "requestbytesrate": 139, "totalresponsebytes": "26482988", "responsebytesrate": 316, "curclntconnections": "8", "surgecount": "0", "cursrvrconnections": "2", "svrestablishedconn": "2", "curreusepool": "2", "maxclients": "0", "curload": "0", "totalconnreassemblyqueue75": "0", "totalconnreassemblyqueueflush": "0", "httpmaxhdrszpkts": "0", "httpmaxhdrfldlenpkts": "0", "tcpmaxooopkts": "0", "curtflags": "0", "totsvrttlbtransactions": "0", "toleratingttlbtransactions": "0", "frustratingttlbtransactions": "0", "vsvrservicehits": "0", "vsvrservicehitsrate": 0, "activetransactions": "0", "serviceorder": "0"}, {"name": "nshttpd-vpn-127.0.0.1-81", "throughput": "0", "throughputrate": 0, "avgsvrttfb": "0", "primaryipaddress": "127.0.0.1", "primaryport": 81, "servicetype": "HTTP", "state": "DOWN", "totalrequests": "0", "requestsrate": 0, "totalresponses": "0", "responsesrate": 0, "totalrequestbytes": "0", "requestbytesrate": 0, "totalresponsebytes": "0", "responsebytesrate": 0, "curclntconnections": "0", "surgecount": "0", "cursrvrconnections": "0", "svrestablishedconn": "0", "curreusepool": "0", "maxclients": "30", "curload": "0", "totalconnreassemblyqueue75": "0", "totalconnreassemblyqueueflush": "0", "httpmaxhdrszpkts": "0", "httpmaxhdrfldlenpkts": "0", "tcpmaxooopkts": "0", "curtflags": "0", "totsvrttlbtransactions": "0", "toleratingttlbtransactions": "0", "frustratingttlbtransactions": "0", "vsvrservicehits": "0", "vsvrservicehitsrate": 0, "activetransactions": "0", "serviceorder": "0"}]}
+  - path: /nitro/v1/stat/lbvserver
+    methods: ['GET']
+    responses:
+      - status_code: 200
+        body: |-
+          {"errorcode":0,"message":"Done","severity":"NONE","lbvserver":[{"name":"elastic","avgcltttlb":"6","cltresponsetimeapdex":1,"vsvrsurgecount":"6","establishedconn":"6","inactsvcs":"6","vslbhealth":"67","primaryipaddress":"8.8.8.8","primaryport":80,"type":"HTTP","state":"DOWN","actsvcs":"10","cpuusagepm":"10","tothits":"10","hitsrate":5,"totalrequests":"5","requestsrate":5,"totalresponses":"5","responsesrate":5,"totalrequestbytes":"7","requestbytesrate":7,"totalresponsebytes":"7","responsebytesrate":7,"totalh2requests":"7","h2requestsrate":7,"totalh2responses":"7","h2responsesrate":7,"totalpktsrecvd":"7","pktsrecvdrate":8,"totalpktssent":"8","pktssentrate":8,"curclntconnections":"8","cursrvrconnections":"8","curpersistencesessions":"8","curbackuppersistencesessions":"8","surgecount":"8","svcsurgecount":"8","sothreshold":"8","totspillovers":"8","labelledconn":"8","pushlabel":"8","deferredreq":"13","deferredreqrate":13,"invalidrequestresponse":"13","invalidrequestresponsedropped":"13","totvserverdownbackuphits":"13","curmptcpsessions":"13","cursubflowconn":"13","totalconnreassemblyqueue75":"13","totalconnreassemblyqueueflush":"3","totalsvrbusyerr":"3","svrbusyerrrate":3,"reqretrycount":"3","reqretrycountexceeded":"3","httpmaxhdrszpkts":"3","httpmaxhdrfldlenpkts":"3","tcpmaxooopkts":"3","totcltttlbtransactions":"3","cltttlbtransactionsrate":3,"toleratingttlbtransactions":"3","toleratingttlbtransactionsrate":1,"frustratingttlbtransactions":"1","frustratingttlbtransactionsrate":1}]}

packages/citrix_adc/_dev/deploy/docker/files/config.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "0.3.0"
+  changes:
+    - description: Citrix ADC integration package with "lbvserver" data stream.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/4418
 - version: "0.2.0"
   changes:
     - description: Citrix ADC integration package with "service" data stream.

packages/citrix_adc/changelog.yml

@@ -0,0 +1,3 @@
+fields:
+  tags:
+    - preserve_original_event

packages/citrix_adc/data_stream/lbvserver/_dev/test/pipeline/test-common-config.yml

@@ -0,0 +1 @@
+{"name":"cpx_default_dns_vserver","avgcltttlb":"4","cltresponsetimeapdex":1,"vsvrsurgecount":"4","establishedconn":"4","inactsvcs":"1","vslbhealth":"67","primaryipaddress":"0.0.0.0","primaryport":0,"type":"DNS","state":"UP","actsvcs":"2","cpuusagepm":"10","tothits":"10","hitsrate":10,"totalrequests":"10","requestsrate":10,"totalresponses":"2","responsesrate":2,"totalrequestbytes":"2","requestbytesrate":2,"totalresponsebytes":"6","responsebytesrate":6,"totalh2requests":"6","h2requestsrate":6,"totalh2responses":"7","h2responsesrate":7,"totalpktsrecvd":"7","pktsrecvdrate":7,"totalpktssent":"8","pktssentrate":8,"curclntconnections":"8","cursrvrconnections":"8","curpersistencesessions":"9","curbackuppersistencesessions":"9","surgecount":"9","svcsurgecount":"9","sothreshold":"11","totspillovers":"11","labelledconn":"11","pushlabel":"11","deferredreq":"1","deferredreqrate":1,"invalidrequestresponse":"1","invalidrequestresponsedropped":"0","totvserverdownbackuphits":"0","curmptcpsessions":"0","cursubflowconn":"0","totalconnreassemblyqueue75":"0","totalconnreassemblyqueueflush":"0","totalsvrbusyerr":"0","svrbusyerrrate":0,"reqretrycount":"0","reqretrycountexceeded":"0","httpmaxhdrszpkts":"0","tcpmaxooopkts":"0","totcltttlbtransactions":"0","cltttlbtransactionsrate":0,"toleratingttlbtransactions":"0","toleratingttlbtransactionsrate":0,"frustratingttlbtransactions":"0","frustratingttlbtransactionsrate":0},{"name":"cpx_default_dns_tcp_vserver","avgcltttlb":"0","cltresponsetimeapdex":1,"vsvrsurgecount":"0","establishedconn":"0","inactsvcs":"1","vslbhealth":"67","primaryipaddress":"0.0.0.0","primaryport":0,"type":"DNS_TCP","state":"UP","actsvcs":"2","cpuusagepm":"0","tothits":"0","hitsrate":0,"totalrequests":"0","requestsrate":0,"totalresponses":"0","responsesrate":0,"totalrequestbytes":"0","requestbytesrate":0,"totalresponsebytes":"0","responsebytesrate":0,"totalh2requests":"0","h2requestsrate":0,"totalh2responses":"0","h2responsesrate":0,"totalpktsrecvd":"0","pktsrecvdrate":0,"totalpktssent":"0","pktssentrate":0,"curclntconnections":"0","cursrvrconnections":"0","curpersistencesessions":"0","curbackuppersistencesessions":"0","surgecount":"0","svcsurgecount":"0","sothreshold":"0","totspillovers":"0","labelledconn":"0","pushlabel":"0","deferredreq":"0","deferredreqrate":0,"invalidrequestresponse":"0","invalidrequestresponsedropped":"0","totvserverdownbackuphits":"0","curmptcpsessions":"0","cursubflowconn":"0","totalconnreassemblyqueue75":"0","totalconnreassemblyqueueflush":"0","totalsvrbusyerr":"0","svrbusyerrrate":0,"reqretrycount":"0","reqretrycountexceeded":"0","httpmaxhdrszpkts":"0","tcpmaxooopkts":"0","totcltttlbtransactions":"0","cltttlbtransactionsrate":0,"toleratingttlbtransactions":"0","toleratingttlbtransactionsrate":0,"frustratingttlbtransactions":"0","frustratingttlbtransactionsrate":0}

packages/citrix_adc/data_stream/lbvserver/_dev/test/pipeline/test-lbvserver-metrics.log

@@ -0,0 +1,139 @@
+{
+    "expected": [
+        {
+            "citrix_adc": {
+                "lbvserver": {
+                    "client": {
+                        "connections": {
+                            "current": {
+                                "count": 8.0
+                            },
+                            "established": {
+                                "count": 4.0
+                            }
+                        },
+                        "response_time": {
+                            "application_performance_index": 1
+                        }
+                    },
+                    "connections": {
+                        "actual": {
+                            "count": 8.0
+                        }
+                    },
+                    "down": {
+                        "backup": {
+                            "hits": 0.0
+                        }
+                    },
+                    "health": 67.0,
+                    "hit": {
+                        "count": 10.0,
+                        "rate": 10
+                    },
+                    "name": "cpx_default_dns_vserver",
+                    "packets": {
+                        "received": {
+                            "count": 7.0
+                        },
+                        "sent": {
+                            "count": 8.0,
+                            "rate": 8
+                        }
+                    },
+                    "protocol": "DNS",
+                    "request": {
+                        "deferred": {
+                            "count": 1.0,
+                            "rate": 1
+                        },
+                        "received": {
+                            "bytes": {
+                                "rate": 2,
+                                "value": 2.0
+                            },
+                            "count": 10.0,
+                            "rate": 10
+                        },
+                        "surge_queue": {
+                            "count": 9.0
+                        },
+                        "waiting": {
+                            "count": 4.0
+                        }
+                    },
+                    "requests_responses": {
+                        "dropped": {
+                            "count": 0.0
+                        },
+                        "invalid": {
+                            "count": 1.0
+                        }
+                    },
+                    "response": {
+                        "received": {
+                            "bytes": {
+                                "rate": 6,
+                                "value": 6.0
+                            },
+                            "count": 2.0,
+                            "rate": 2
+                        }
+                    },
+                    "service": {
+                        "active": {
+                            "count": 2.0
+                        },
+                        "inactive": {
+                            "count": 1.0
+                        }
+                    },
+                    "spillover": {
+                        "count": 11.0
+                    },
+                    "state": "UP",
+                    "threshold": {
+                        "spillover": 11.0
+                    },
+                    "time_to_last_byte": {
+                        "avg": 4.0
+                    },
+                    "transaction": {
+                        "frustrating": {
+                            "count": 0.0
+                        },
+                        "tolerable": {
+                            "count": 0.0
+                        }
+                    }
+                }
+            },
+            "ecs": {
+                "version": "8.4.0"
+            },
+            "event": {
+                "category": [
+                    "web"
+                ],
+                "kind": "event",
+                "module": "citrix_adc",
+                "original": "{\"name\":\"cpx_default_dns_vserver\",\"avgcltttlb\":\"4\",\"cltresponsetimeapdex\":1,\"vsvrsurgecount\":\"4\",\"establishedconn\":\"4\",\"inactsvcs\":\"1\",\"vslbhealth\":\"67\",\"primaryipaddress\":\"0.0.0.0\",\"primaryport\":0,\"type\":\"DNS\",\"state\":\"UP\",\"actsvcs\":\"2\",\"cpuusagepm\":\"10\",\"tothits\":\"10\",\"hitsrate\":10,\"totalrequests\":\"10\",\"requestsrate\":10,\"totalresponses\":\"2\",\"responsesrate\":2,\"totalrequestbytes\":\"2\",\"requestbytesrate\":2,\"totalresponsebytes\":\"6\",\"responsebytesrate\":6,\"totalh2requests\":\"6\",\"h2requestsrate\":6,\"totalh2responses\":\"7\",\"h2responsesrate\":7,\"totalpktsrecvd\":\"7\",\"pktsrecvdrate\":7,\"totalpktssent\":\"8\",\"pktssentrate\":8,\"curclntconnections\":\"8\",\"cursrvrconnections\":\"8\",\"curpersistencesessions\":\"9\",\"curbackuppersistencesessions\":\"9\",\"surgecount\":\"9\",\"svcsurgecount\":\"9\",\"sothreshold\":\"11\",\"totspillovers\":\"11\",\"labelledconn\":\"11\",\"pushlabel\":\"11\",\"deferredreq\":\"1\",\"deferredreqrate\":1,\"invalidrequestresponse\":\"1\",\"invalidrequestresponsedropped\":\"0\",\"totvserverdownbackuphits\":\"0\",\"curmptcpsessions\":\"0\",\"cursubflowconn\":\"0\",\"totalconnreassemblyqueue75\":\"0\",\"totalconnreassemblyqueueflush\":\"0\",\"totalsvrbusyerr\":\"0\",\"svrbusyerrrate\":0,\"reqretrycount\":\"0\",\"reqretrycountexceeded\":\"0\",\"httpmaxhdrszpkts\":\"0\",\"tcpmaxooopkts\":\"0\",\"totcltttlbtransactions\":\"0\",\"cltttlbtransactionsrate\":0,\"toleratingttlbtransactions\":\"0\",\"toleratingttlbtransactionsrate\":0,\"frustratingttlbtransactions\":\"0\",\"frustratingttlbtransactionsrate\":0},{\"name\":\"cpx_default_dns_tcp_vserver\",\"avgcltttlb\":\"0\",\"cltresponsetimeapdex\":1,\"vsvrsurgecount\":\"0\",\"establishedconn\":\"0\",\"inactsvcs\":\"1\",\"vslbhealth\":\"67\",\"primaryipaddress\":\"0.0.0.0\",\"primaryport\":0,\"type\":\"DNS_TCP\",\"state\":\"UP\",\"actsvcs\":\"2\",\"cpuusagepm\":\"0\",\"tothits\":\"0\",\"hitsrate\":0,\"totalrequests\":\"0\",\"requestsrate\":0,\"totalresponses\":\"0\",\"responsesrate\":0,\"totalrequestbytes\":\"0\",\"requestbytesrate\":0,\"totalresponsebytes\":\"0\",\"responsebytesrate\":0,\"totalh2requests\":\"0\",\"h2requestsrate\":0,\"totalh2responses\":\"0\",\"h2responsesrate\":0,\"totalpktsrecvd\":\"0\",\"pktsrecvdrate\":0,\"totalpktssent\":\"0\",\"pktssentrate\":0,\"curclntconnections\":\"0\",\"cursrvrconnections\":\"0\",\"curpersistencesessions\":\"0\",\"curbackuppersistencesessions\":\"0\",\"surgecount\":\"0\",\"svcsurgecount\":\"0\",\"sothreshold\":\"0\",\"totspillovers\":\"0\",\"labelledconn\":\"0\",\"pushlabel\":\"0\",\"deferredreq\":\"0\",\"deferredreqrate\":0,\"invalidrequestresponse\":\"0\",\"invalidrequestresponsedropped\":\"0\",\"totvserverdownbackuphits\":\"0\",\"curmptcpsessions\":\"0\",\"cursubflowconn\":\"0\",\"totalconnreassemblyqueue75\":\"0\",\"totalconnreassemblyqueueflush\":\"0\",\"totalsvrbusyerr\":\"0\",\"svrbusyerrrate\":0,\"reqretrycount\":\"0\",\"reqretrycountexceeded\":\"0\",\"httpmaxhdrszpkts\":\"0\",\"tcpmaxooopkts\":\"0\",\"totcltttlbtransactions\":\"0\",\"cltttlbtransactionsrate\":0,\"toleratingttlbtransactions\":\"0\",\"toleratingttlbtransactionsrate\":0,\"frustratingttlbtransactions\":\"0\",\"frustratingttlbtransactionsrate\":0}",
+                "type": [
+                    "info"
+                ]
+            },
+            "related": {
+                "ip": [
+                    "0.0.0.0"
+                ]
+            },
+            "server": {
+                "ip": "0.0.0.0",
+                "port": 0
+            },
+            "tags": [
+                "preserve_original_event"
+            ]
+        }
+    ]
+}

packages/citrix_adc/data_stream/lbvserver/_dev/test/pipeline/test-lbvserver-metrics.log-expected.json

@@ -0,0 +1,9 @@
+input: httpjson
+service: citrix_adc
+vars:
+  hostname: http://{{Hostname}}:{{Port}}
+  username: nsroot
+  password: nsroot
+data_stream:
+  vars:
+    preserve_original_event: true

packages/citrix_adc/data_stream/lbvserver/_dev/test/system/test-default-config.yml

@@ -0,0 +1,29 @@
+config_version: 2
+interval: {{period}}
+{{#if username}}
+{{#if password}}
+auth.basic.user: {{username}}
+auth.basic.password: {{password}}
+{{/if}}
+{{/if}}
+{{#if ssl}}
+request.ssl: {{ssl}}
+{{/if}}
+request.method: GET
+request.url: {{hostname}}/nitro/v1/stat/lbvserver
+response.split:
+  target: body.lbvserver
+tags:
+{{#if preserve_original_event}}
+  - preserve_original_event
+{{/if}}
+{{#each tags as |tag|}}
+  - {{tag}}
+{{/each}}
+{{#contains "forwarded" tags}}
+publisher_pipeline.disable_host: true
+{{/contains}}
+{{#if processors}}
+processors:
+{{processors}}
+{{/if}}