CP-8403 Adding Telegraf-based metric collection.

[scottmdlpx]

Initial addition of configuration and control files to enable performance metric collection using the Telegraf agent.
See also IDEA-2835 : Improving Support Bundle Performance Metrics

Includes:

• Service definition and startup script for "delphix-telegraf"
• Modified version of "estat" adding JSON output via a "-j" option
• A "perf_playbook" wrapper script to enable/disable enhanced collection
• Configuration file sections (combined on startup)
• Simple wrappers to facilitate parsing of "nfs_threads", "zpool iostat -o",
  and "zcache stats -a" outputs

The service starts with a "base" set of metrics, but will include Object Storage
metrics when it is detected, and will include Performance Playbook commands
if that has been enabled (manually). The config is reassembled each startup.

File paths intended:

/opt/delphix/server/bin/delphix-telegraf-service
/lib/systemd/system/delphix-telegraf.service
/usr/bin/estat
/etc/telegraf/nfs-threads.sh
/opt/delphix/server/bin/perf_playbook
/etc/telegraf/telegraf.base
/etc/telegraf/telegraf.inputs.dose
/etc/telegraf/telegraf.inputs.playbook
/etc/telegraf/zcache-stats.sh
/etc/telegraf/zpool-iostat-o.sh

This configuration records 3 output files (rotated on size) for main metrics,
aggregate statistics (min,max,mean,stddev) and Playbook outputs to enable
independent retention periods.

[jwk404]

Very happy to see this. Thanks for adding it.

[scottmdlpx]

ab-pre-push Success:
http://selfservice.jenkins.delphix.com/job/appliance-build-orchestrator-pre-push/2284/

Cloned a VM from the build and:

$ sudo systemctl status delphix-telegraf
● delphix-telegraf.service - Delphix Telegraf Metric Collection Agent
     Loaded: loaded (/lib/systemd/system/delphix-telegraf.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-06-27 08:18:48 UTC; 4h 13min ago
       Docs: https://github.com/influxdata/telegraf
   Main PID: 4201 (delphix-telegra)
      Tasks: 12 (limit: 8751)
     Memory: 91.5M
     CGroup: /system.slice/delphix-telegraf.service
             ├─4201 /bin/bash /usr/bin/delphix-telegraf-service
             └─4472 /usr/bin/telegraf -config /etc/telegraf/telegraf.conf

Output files are being logged under /var/log/telegraf:

$ pwd
/var/log/telegraf
$ ls
metric_aggregates.json                        metrics_estat.json
metrics.json                                  metrics_zfs.json

I also pulled a Support Bundle and verified these are included.

Enabling the performance playbook scripts:

$ sudo perf_playbook enable
Mon 27 Jun 2022 12:32:56 PM UTC
Enabling Performance Playbook Metrics
delphix@ip-10-110-236-17:~$ sudo systemctl status delphix-telegraf
● delphix-telegraf.service - Delphix Telegraf Metric Collection Agent
     Loaded: loaded (/lib/systemd/system/delphix-telegraf.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-06-27 12:32:56 UTC; 3s ago
       Docs: https://github.com/influxdata/telegraf
   Main PID: 40001 (delphix-telegra)
      Tasks: 29 (limit: 8751)
     Memory: 282.2M
     CGroup: /system.slice/delphix-telegraf.service
             ├─40001 /bin/bash /usr/bin/delphix-telegraf-service
             ├─40117 /usr/bin/telegraf -config /etc/telegraf/telegraf.conf
             ├─40124 python3 /usr/bin/estat nfs -jm 10
             ├─40125 python3 /usr/bin/estat iscsi -jm 10
             ├─40126 python3 /usr/bin/estat zpl -jm 10
             ├─40127 python3 /usr/bin/estat backend-io -jm 10
             ├─40130 python3 /usr/bin/estat zvol -jm 10
             ├─40131 python3 /usr/bin/estat zio -jm 10
             ├─40135 /bin/sh /etc/telegraf/nfs-threads.sh
             ├─40136 python3 /usr/bin/nfs_threads
             └─40137 grep -E --line-buffered -v thr

and disabling them again:

$ sudo perf_playbook disable
Mon 27 Jun 2022 12:33:09 PM UTC
Disabling Performance Playbook Metrics
$ sudo systemctl status delphix-telegraf
● delphix-telegraf.service - Delphix Telegraf Metric Collection Agent
     Loaded: loaded (/lib/systemd/system/delphix-telegraf.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-06-27 12:33:09 UTC; 3s ago
       Docs: https://github.com/influxdata/telegraf
   Main PID: 40215 (delphix-telegra)
      Tasks: 9 (limit: 8751)
     Memory: 24.0M
     CGroup: /system.slice/delphix-telegraf.service
             ├─40215 /bin/bash /usr/bin/delphix-telegraf-service
             └─40323 /usr/bin/telegraf -config /etc/telegraf/telegraf.conf

I also tested streaming to InfluxDB directly by uncommenting the relevant config and this works.