Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[vSphere][virtualmachine] Add support for additional metrics #10942

Merged
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/vsphere/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.14.0-next"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets update this to latest 1.15.0-next

changes:
- description: Add additional metrics for virtualmachine datastream
type: enhancement
link: https://github.com/elastic/integrations/pull/10942
- version: "1.14.0"
changes:
- description: Add support for processors in datastore, host and virtualmachine.
Expand Down
116 changes: 98 additions & 18 deletions packages/vsphere/data_stream/virtualmachine/fields/fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,60 +2,140 @@
type: group
description: virtualmachine
fields:
- name: host.id
type: keyword
description: Id of the host hosting the virtualmachine
- name: host.hostname
type: keyword
description: Name of the host hosting the virtualmachine
- name: host
type: group
fields:
- name: id
type: keyword
description: >
Host id.
- name: hostname
type: keyword
description: >
Hostname of the host.
- name: name
type: keyword
# Reason to add as a dimension field: Each VM has Unique name assigned
# Reason to add as a dimension field: Each VM has Unique name assigned.
dimension: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lucian-ioan : I suppose you are using the fields.yml directly from the beats code.
That would not be correct as we don't consider tsdb there.
Also, for any new fields please keep TSDB into consideration.

description: Virtual Machine name
description: >
Virtual Machine name.
- name: os
type: keyword
description: Virtual Machine Operating System name
description: >
Virtual Machine Operating System name.
- name: cpu.used.mhz
type: long
metric_type: gauge
description: Used CPU of virtualmachine in Mhz
description: >
Used CPU in Mhz.
- name: cpu.total.mhz
type: long
metric_type: counter
description: Total CPU of virtualmachine in Mhz
description: >
Total Reserved CPU in Mhz.
- name: cpu.free.mhz
type: long
metric_type: gauge
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to remove these metric_types?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, reverting, tested first with beats code directly.

description: Available CPU of virtualmachine in Mhz
description: >
Available CPU in Mhz.
- name: memory.used.guest.bytes
type: long
metric_type: gauge
unit: byte
description: Used Memory of Guest in bytes
description: >
Used Memory of Guest in bytes.
format: bytes
- name: memory.used.host.bytes
type: long
metric_type: gauge
unit: byte
description: Used Memory of Host in bytes
description: >
Used Memory of Host in bytes.
format: bytes
- name: memory.total.guest.bytes
type: long
metric_type: gauge
unit: byte
description: Total Memory of Guest in bytes
description: >
Total Memory of Guest in bytes.
format: bytes
- name: memory.free.guest.bytes
type: long
metric_type: gauge
unit: byte
description: Free Memory of Guest in bytes
description: >
Free Memory of Guest in bytes.
format: bytes
- name: custom_fields
type: object
object_type: keyword
description: Custom fields
description: >
Custom fields.
- name: network_names
type: keyword
description: Network names
description: >
Network names.
- name: datastore
type: group
fields:
- name: names
type: keyword
description: >
Names of the datastore associated to this virtualmachine.
- name: count
type: long
metric_type: gauge
description: >
Number of datastores associated to this virtualmachine.
- name: network
type: group
fields:
- name: names
type: keyword
description: >
Names of the networks associated to this virtualmachine.
- name: count
type: long
metric_type: gauge
description: >
Number of networks associated to this virtualmachine.
- name: status
type: keyword
description: >
Overall health and status of a virtual machine.
- name: uptime
type: long
metric_type: counter
description: >
The uptime of the VM in seconds.
- name: snapshot
type: group
fields:
- name: info
type: group
description: Details of the snapshots of this virtual machine.
fields:
- name: id
type: long
description: Snapshot ID.
- name: name
type: keyword
description: Snapshot name.
- name: description
type: keyword
description: Snapshot description.
- name: createtime
type: date
description: Snapshot creation time.
- name: state
type: keyword
description: Snapshot state (e.g., poweredOn).
- name: count
type: long
metric_type: gauge
description: The number of snapshots of this virtualmachine.
- name: triggerd_alarms.*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to incorporate the latest change of this name change yesterday.
"triggered_alamrs"

type: object
object_type: keyword
description: >-
List of all the triggerd alarms.
13 changes: 13 additions & 0 deletions packages/vsphere/data_stream/virtualmachine/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,19 @@ streams:
type: bool
multi: false
default: false
- name: period
type: text
title: Period
description: The vSphere performance API allows for collecting host and datastore metrics at various intervals, including real-time (every 20 seconds), 5 minutes, or longer durations. Properly configuring the "period" parameter is crucial to manage usage restrictions and ensure accurate data collection. For more information on optimizing your data collection setup, refer to the [Data Collection Intervals](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-247646EA-A04B-411A-8DD4-62A3DCFCF49B.html) and [Data Collection Levels](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.monitoring.doc/GUID-25800DE4-68E5-41CC-82D9-8811E27924BC.html) documentation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change the description as discussed yesterday

default: 20s
- name: tags
type: text
title: Tags
multi: true
required: true
show_user: false
default:
- vsphere-virtualmachine
- name: processors
type: yaml
title: Processors
Expand Down
112 changes: 78 additions & 34 deletions packages/vsphere/data_stream/virtualmachine/sample_event.json
Original file line number Diff line number Diff line change
@@ -1,48 +1,50 @@
{
"@timestamp": "2023-06-29T08:06:40.827Z",
"@timestamp": "2024-09-02T09:44:14.128Z",
"agent": {
"ephemeral_id": "527dd76f-fe04-4478-b02c-110b5f47ccf4",
"id": "5096d7cc-1e4b-4959-abea-7355be2913a7",
"name": "docker-fleet-agent",
"ephemeral_id": "2d60906e-b972-4981-a356-c57ccb79108b",
"id": "8ea73fb3-a0a0-4270-aad6-e88edb8b385a",
"name": "elastic-agent-55444",
"type": "metricbeat",
"version": "8.8.1"
"version": "8.16.0"
},
"data_stream": {
"dataset": "vsphere.virtualmachine",
"namespace": "ep",
"namespace": "64133",
"type": "metrics"
},
"ecs": {
"version": "8.11.0"
},
"elastic_agent": {
"id": "5096d7cc-1e4b-4959-abea-7355be2913a7",
"snapshot": false,
"version": "8.8.1"
"id": "8ea73fb3-a0a0-4270-aad6-e88edb8b385a",
"snapshot": true,
"version": "8.16.0"
},
"event": {
"agent_id_status": "verified",
"dataset": "vsphere.virtualmachine",
"duration": 14355583,
"ingested": "2023-06-29T08:06:41Z",
"duration": 52726937,
"ingested": "2024-09-02T09:44:17Z",
"module": "vsphere"
},
"host": {
"architecture": "aarch64",
"containerized": false,
"hostname": "docker-fleet-agent",
"id": "d08b346fbb8f49f5a2bb1a477f8ceb54",
"architecture": "x86_64",
"containerized": true,
"hostname": "elastic-agent-55444",
"id": "e744630f9d4f43dc818e497d221bd0b2",
"ip": [
"172.23.0.7"
"172.18.0.4",
"172.20.0.2"
],
"mac": [
"02-42-AC-17-00-07"
"02-42-AC-12-00-04",
"02-42-AC-14-00-02"
],
"name": "docker-fleet-agent",
"name": "elastic-agent-55444",
"os": {
"codename": "focal",
"family": "debian",
"kernel": "5.10.104-linuxkit",
"kernel": "5.15.153.1-microsoft-standard-WSL2",
"name": "Ubuntu",
"platform": "ubuntu",
"type": "linux",
Expand All @@ -54,42 +56,84 @@
"period": 10000
},
"service": {
"address": "https://elastic-package-service_vsphere-metrics_1:8989/sdk",
"address": "https://svc-vsphere-metrics:8989/sdk",
"type": "vsphere"
},
"vsphere": {
"virtualmachine": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these events autogenerated by system tests ?
Looks like these are being used from beats code ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested with beats code, regenerated it now.

"name": "xt0nmfpv9",
"uptime": 5348978,
"status": "green",
"host": {
"id": "host-32",
"hostname": "phx-w1c1-esxi04.com"
},
"cpu": {
"free": {
"mhz": 0
},
"used": {
"mhz": 161
},
"total": {
"mhz": 0
}
},
"host": {
"hostname": "DC0_H0",
"id": "host-21"
"network": {
"names": [
"PROD_VCF_VMS"
],
"count": 1
},
"memory": {
"free": {
"used": {
"guest": {
"bytes": 33554432
"bytes": 686817280
},
"host": {
"bytes": 29027729408
}
},
"total": {
"guest": {
"bytes": 33554432
"bytes": 68719476736
}
},
"used": {
"free": {
"guest": {
"bytes": 0
},
"host": {
"bytes": 0
"bytes": 68032659456
}
}
},
"name": "DC0_H0_VM0",
"os": "otherGuest"
"network_names": [
"PROD_VCF_VMS"
],
"datastore": {
"count": 1,
"names": [
"VxRailtoup-Virtual-Datastore-bc1d-5aa310fb"
]
},
"os": "CentOS 4/5/6/7 (64-bit)",
"snapshot": {
"info": [
{
"id": 1,
"name": "VM Snapshot 7%2f3%2f2024, 4:01:21 PM",
"description": "Created to demo",
"createtime": "2024-07-03T20:01:34.329Z",
"state": "poweredOn"
},
{
"createtime": "2024-07-05T23:35:40.859Z",
"state": "poweredOn",
"id": 2,
"name": "VM Snapshot 7%2f5%2f2024, 7:35:37 PM",
"description": "backup"
}
],
"count": 2
}
}
}
}
}
Loading