Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[vSphere][virtualmachine] Add support for additional metrics #10942

Merged
merged 23 commits into from
Sep 20, 2024
Merged

[vSphere][virtualmachine] Add support for additional metrics #10942

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
8e782c6
add additional fields
lucian-ioan Aug 30, 2024
3e86570
revert fields metric types
lucian-ioan Sep 2, 2024
260ef06
redo system test
lucian-ioan Sep 2, 2024
953b653
add changelog entry
lucian-ioan Sep 2, 2024
e1af4af
add snapshot field
lucian-ioan Sep 18, 2024
56c61bd
use group for host
lucian-ioan Sep 18, 2024
28978e0
update info field
lucian-ioan Sep 18, 2024
6c72eb4
add triggered alarms
lucian-ioan Sep 18, 2024
9d3b1a0
update manifest file
lucian-ioan Sep 18, 2024
3f909da
Merge branch 'main' into update_vsphere_virtual_machine_integration
lucian-ioan Sep 18, 2024
a782210
add more metric types
lucian-ioan Sep 18, 2024
8e01de8
change snapshot info to group
lucian-ioan Sep 18, 2024
d6b9916
update readme
lucian-ioan Sep 18, 2024
eca6813
update type
lucian-ioan Sep 18, 2024
649c864
default period description
lucian-ioan Sep 19, 2024
d2f467e
update version to 1.15.0-next
lucian-ioan Sep 19, 2024
6b27dc4
update triggered alarms
lucian-ioan Sep 19, 2024
f546aa8
add system tests
lucian-ioan Sep 19, 2024
9c318ec
Merge remote-tracking branch 'origin' into update_vsphere_virtual_mac…
lucian-ioan Sep 19, 2024
75b3a08
update readme
lucian-ioan Sep 19, 2024
4aa0c34
add tag in stream.yml.hbs
lucian-ioan Sep 19, 2024
ba0636d
update IPs and domains
lucian-ioan Sep 20, 2024
a009454
Merge branch 'main' into update_vsphere_virtual_machine_integration
lucian-ioan Sep 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 57 additions & 26 deletions packages/vsphere/data_stream/virtualmachine/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,58 +4,89 @@
fields:
- name: host.id
type: keyword
description: Id of the host hosting the virtualmachine
description: >
Host id.
- name: host.hostname
type: keyword
description: Name of the host hosting the virtualmachine
description: >
Hostname of the host.
- name: name
type: keyword
# Reason to add as a dimension field: Each VM has Unique name assigned
dimension: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lucian-ioan : I suppose you are using the fields.yml directly from the beats code.
That would not be correct as we don't consider tsdb there.
Also, for any new fields please keep TSDB into consideration.

description: Virtual Machine name
# Reason to add as a dimension field: Each VM has Unique name assigned.
description: >
Virtual Machine name.
- name: os
type: keyword
description: Virtual Machine Operating System name
description: >
Virtual Machine Operating System name.
- name: cpu.used.mhz
type: long
metric_type: gauge
description: Used CPU of virtualmachine in Mhz
description: >
Used CPU in Mhz.
- name: cpu.total.mhz
type: long
metric_type: counter
description: Total CPU of virtualmachine in Mhz
description: >
Total CPU in Mhz.
- name: cpu.free.mhz
type: long
metric_type: gauge
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to remove these metric_types?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, reverting, tested first with beats code directly.

description: Available CPU of virtualmachine in Mhz
description: >
Available CPU in Mhz.
- name: memory.used.guest.bytes
type: long
metric_type: gauge
unit: byte
description: Used Memory of Guest in bytes
description: >
Used Memory of Guest in bytes.
format: bytes
- name: memory.used.host.bytes
type: long
metric_type: gauge
unit: byte
description: Used Memory of Host in bytes
description: >
Used Memory of Host in bytes.
format: bytes
- name: memory.total.guest.bytes
type: long
metric_type: gauge
unit: byte
description: Total Memory of Guest in bytes
description: >
Total Memory of Guest in bytes.
format: bytes
- name: memory.free.guest.bytes
type: long
metric_type: gauge
unit: byte
description: Free Memory of Guest in bytes
description: >
Free Memory of Guest in bytes.
format: bytes
- name: custom_fields
type: object
object_type: keyword
description: Custom fields
description: >
Custom fields.
- name: network_names
type: keyword
description: Network names
description: >
Network names.
- name: datastore
type: group
fields:
- name: names
type: keyword
description: >
Names of the datastore associated to this virtualmachine.
- name: count
type: long
description: >
Number of datastores associated to this virtualmachine.
- name: network
type: group
fields:
- name: names
type: keyword
description: >
Names of the networks associated to this virtualmachine.
- name: count
type: long
description: >
Number of networks associated to this virtualmachine.
- name: status
type: keyword
description: >
Overall health and status of a virtual machine.
- name: uptime
type: long
description: >
The uptime of the VM in seconds.
57 changes: 39 additions & 18 deletions packages/vsphere/data_stream/virtualmachine/sample_event.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,37 +59,58 @@
},
"vsphere": {
"virtualmachine": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these events autogenerated by system tests ?
Looks like these are being used from beats code ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested with beats code, regenerated it now.

"cpu": {
"used": {
"mhz": 0
}
},
"host": {
"hostname": "DC0_H0",
"id": "host-21"
"network.names": [
"PROD_VCF"
],
"os": "Microsoft Windows 10 (64-bit)",
"datastore": {
"count": 1
},
"memory": {
"free": {
"used": {
"guest": {
"bytes": 33554432
"bytes": 0
},
"host": {
"bytes": 0
}
},
"total": {
"guest": {
"bytes": 33554432
"bytes": 4294967296
}
},
"used": {
"free": {
"guest": {
"bytes": 0
},
"host": {
"bytes": 0
"bytes": 4294967296
}
}
},
"name": "DC0_H0_VM0",
"os": "otherGuest"
"network": {
"count": 1
},
"host.hostname": "phx-w1c1",
"name": "phx-14",
"cpu": {
"free": {
"mhz": 0
},
"used": {
"mhz": 0
},
"total": {
"mhz": 0
}
},
"uptime": 0,
"status": "green",
"network_names": [
"PROD_VCF"
],
"datastore.names": [
"VxRailtec-Virtual-SAN-Datastore-247df-bc1d-5aad2"
],
"host.id": "host-20"
}
}
}
129 changes: 78 additions & 51 deletions packages/vsphere/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,37 +79,58 @@ An example event for `virtualmachine` looks as following:
},
"vsphere": {
"virtualmachine": {
"cpu": {
"used": {
"mhz": 0
}
},
"host": {
"hostname": "DC0_H0",
"id": "host-21"
"network.names": [
"PROD_VCF"
],
"os": "Microsoft Windows 10 (64-bit)",
"datastore": {
"count": 1
},
"memory": {
"free": {
"used": {
"guest": {
"bytes": 33554432
"bytes": 0
},
"host": {
"bytes": 0
}
},
"total": {
"guest": {
"bytes": 33554432
"bytes": 4294967296
}
},
"used": {
"free": {
"guest": {
"bytes": 0
},
"host": {
"bytes": 0
"bytes": 4294967296
}
}
},
"name": "DC0_H0_VM0",
"os": "otherGuest"
"network": {
"count": 1
},
"host.hostname": "phx-w1c1",
"name": "phx-14",
"cpu": {
"free": {
"mhz": 0
},
"used": {
"mhz": 0
},
"total": {
"mhz": 0
}
},
"uptime": 0,
"status": "green",
"network_names": [
"PROD_VCF"
],
"datastore.names": [
"VxRailtec-Virtual-SAN-Datastore-247df-bc1d-5aad2"
],
"host.id": "host-20"
}
}
}
Expand All @@ -121,39 +142,45 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur

**Exported fields**

| Field | Description | Type | Unit | Metric Type |
|---|---|---|---|---|
| @timestamp | Event timestamp. | date | | |
| agent.id | | keyword | | |
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | |
| cloud.availability_zone | Availability zone in which this host, resource, or service is located. | keyword | | |
| cloud.instance.id | Instance ID of the host machine. | keyword | | |
| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | |
| cloud.region | Region in which this host, resource, or service is located. | keyword | | |
| container.id | Unique container id. | keyword | | |
| data_stream.dataset | Data stream dataset. | constant_keyword | | |
| data_stream.namespace | Data stream namespace. | constant_keyword | | |
| data_stream.type | Data stream type. | constant_keyword | | |
| event.dataset | Event dataset | constant_keyword | | |
| event.module | Event module | constant_keyword | | |
| host.containerized | If the host is a container. | boolean | | |
| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | |
| host.os.build | OS build information. | keyword | | |
| host.os.codename | OS codename, if any. | keyword | | |
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | |
| vsphere.virtualmachine.cpu.free.mhz | Available CPU of virtualmachine in Mhz | long | | gauge |
| vsphere.virtualmachine.cpu.total.mhz | Total CPU of virtualmachine in Mhz | long | | counter |
| vsphere.virtualmachine.cpu.used.mhz | Used CPU of virtualmachine in Mhz | long | | gauge |
| vsphere.virtualmachine.custom_fields | Custom fields | object | | |
| vsphere.virtualmachine.host.hostname | Name of the host hosting the virtualmachine | keyword | | |
| vsphere.virtualmachine.host.id | Id of the host hosting the virtualmachine | keyword | | |
| vsphere.virtualmachine.memory.free.guest.bytes | Free Memory of Guest in bytes | long | byte | gauge |
| vsphere.virtualmachine.memory.total.guest.bytes | Total Memory of Guest in bytes | long | byte | gauge |
| vsphere.virtualmachine.memory.used.guest.bytes | Used Memory of Guest in bytes | long | byte | gauge |
| vsphere.virtualmachine.memory.used.host.bytes | Used Memory of Host in bytes | long | byte | gauge |
| vsphere.virtualmachine.name | Virtual Machine name | keyword | | |
| vsphere.virtualmachine.network_names | Network names | keyword | | |
| vsphere.virtualmachine.os | Virtual Machine Operating System name | keyword | | |
| Field | Description | Type |
|---|---|---|
| @timestamp | Event timestamp. | date |
| agent.id | | keyword |
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword |
| cloud.availability_zone | Availability zone in which this host, resource, or service is located. | keyword |
| cloud.instance.id | Instance ID of the host machine. | keyword |
| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword |
| cloud.region | Region in which this host, resource, or service is located. | keyword |
| container.id | Unique container id. | keyword |
| data_stream.dataset | Data stream dataset. | constant_keyword |
| data_stream.namespace | Data stream namespace. | constant_keyword |
| data_stream.type | Data stream type. | constant_keyword |
| event.dataset | Event dataset | constant_keyword |
| event.module | Event module | constant_keyword |
| host.containerized | If the host is a container. | boolean |
| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword |
| host.os.build | OS build information. | keyword |
| host.os.codename | OS codename, if any. | keyword |
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword |
| vsphere.virtualmachine.cpu.free.mhz | Available CPU in Mhz. | long |
| vsphere.virtualmachine.cpu.total.mhz | Total CPU in Mhz. | long |
| vsphere.virtualmachine.cpu.used.mhz | Used CPU in Mhz. | long |
| vsphere.virtualmachine.custom_fields | Custom fields. | object |
| vsphere.virtualmachine.datastore.count | Number of datastores associated to this virtualmachine. | long |
| vsphere.virtualmachine.datastore.names | Names of the datastore associated to this virtualmachine. | keyword |
| vsphere.virtualmachine.host.hostname | Hostname of the host. | keyword |
| vsphere.virtualmachine.host.id | Host id. | keyword |
| vsphere.virtualmachine.memory.free.guest.bytes | Free Memory of Guest in bytes. | long |
| vsphere.virtualmachine.memory.total.guest.bytes | Total Memory of Guest in bytes. | long |
| vsphere.virtualmachine.memory.used.guest.bytes | Used Memory of Guest in bytes. | long |
| vsphere.virtualmachine.memory.used.host.bytes | Used Memory of Host in bytes. | long |
| vsphere.virtualmachine.name | Virtual Machine name. | keyword |
| vsphere.virtualmachine.network.count | Number of networks associated to this virtualmachine. | long |
| vsphere.virtualmachine.network.names | Names of the networks associated to this virtualmachine. | keyword |
| vsphere.virtualmachine.network_names | Network names. | keyword |
| vsphere.virtualmachine.os | Virtual Machine Operating System name. | keyword |
| vsphere.virtualmachine.status | Overall health and status of a virtual machine. | keyword |
| vsphere.virtualmachine.uptime | The uptime of the VM in seconds. | long |


### Host Metrics
Expand Down