Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cluster multi ESX LiteStats -null #342

Closed
n0405 opened this issue Apr 6, 2023 · 23 comments
Closed

Cluster multi ESX LiteStats -null #342

n0405 opened this issue Apr 6, 2023 · 23 comments
Labels

Comments

@n0405
Copy link

n0405 commented Apr 6, 2023

Hi ,

In VMware Cluster Multi ESX LiteStats for several clusters there is no data , I did migrate from older version to new version of sexygraf. And I can see older data and new data for some cluster , but not for all cluster . FOr two cluster I can see only old data .
Quering the table i find that date is missing . whisper-fetch.py /mnt/wfs/whisper/vmw/vcsa1/DC1/cluster1/esx/esxi1/fatstats/overallCpuUtilization.wsp
1680766200 None
1680766500 None
1680766800 None
1680767100 None
1680767400 None
1680767700 None
1680768000 None
1680768300 None
1680768600 None
1680768900 None

graf litestat
image

I can't find the reason why I'm missing this metric

@rschitz
Copy link
Member

rschitz commented Apr 7, 2023

Hi, can you share logs please?

@n0405
Copy link
Author

n0405 commented Apr 7, 2023

Hi,
which logs are you referring to? some specific log ?

PowerShell transcript start
Start time: 20230407120302


Transcript started, output file is /var/log/sexigraf/ViOfflineInventory.log
2023-04-07T12:03:02.2224834+02:00 [INFO] ViOfflineInventory v0.9.68
2023-04-07T12:03:02.2284901+02:00 [INFO] Importing PowerCli and Graphite PowerShell modules ...
2023-04-07T12:03:04.8203700+02:00 [INFO] Looking for another ViOfflineInventory ...
2023-04-07T12:03:04.8404506+02:00 [INFO] VI servers listing ...
2023-04-07T12:03:04.8598055+02:00 [INFO] SessionToken found in SessionFile, attempting connection to vcsa1 ...
2023-04-07T12:03:07.5166296+02:00 [INFO] Connected to vCenter vcsa1 version 7.0.3 build 20990077
2023-04-07T12:03:07.5256937+02:00 [INFO] Start processing vCenter/ESX vcsa1 ...
2023-04-07T12:03:07.6157085+02:00 [INFO] Collecting objects in vcsa1 ...
2023-04-07T12:03:09.1047863+02:00 [INFO] Building hashtables ...
2023-04-07T12:03:09.8198953+02:00 [INFO] SessionToken found in SessionFile, attempting connection to vcsa2 ...
2023-04-07T12:03:10.5383287+02:00 [INFO] Connected to vCenter vcsa2 version 7.0.3 build 20990077
2023-04-07T12:03:10.5404018+02:00 [INFO] Start processing vCenter/ESX vcsa2 ...
2023-04-07T12:03:10.5876557+02:00 [INFO] Collecting objects in vcsa2 ...
2023-04-07T12:03:11.9180958+02:00 [INFO] Building hashtables ...
2023-04-07T12:03:12.3039491+02:00 [INFO] SessionToken found in SessionFile, attempting connection to vcsa3 ...
2023-04-07T12:03:13.0113482+02:00 [INFO] Connected to vCenter vcsa3 version 6.7.0 build 20504362
2023-04-07T12:03:13.0121650+02:00 [INFO] Start processing vCenter/ESX vcsa3 ...
2023-04-07T12:03:13.0700234+02:00 [INFO] Collecting objects in vcsa3 ...
2023-04-07T12:03:14.0340733+02:00 [INFO] Building hashtables ...
2023-04-07T12:03:14.1304752+02:00 [INFO] SessionToken found in SessionFile, attempting connection to vcsa4 ...
2023-04-07T12:03:14.7532818+02:00 [INFO] Connected to vCenter vcsa4 version 7.0.3 build 20990077
2023-04-07T12:03:14.7541100+02:00 [INFO] Start processing vCenter/ESX vcsa4 ...
2023-04-07T12:03:14.8012202+02:00 [INFO] Collecting objects in vcsa4 ...
2023-04-07T12:03:16.3757703+02:00 [INFO] Building hashtables ...
2023-04-07T12:03:16.9734979+02:00 [INFO] SessionToken found in SessionFile, attempting connection to vcsa5...
2023-04-07T12:03:17.7052455+02:00 [INFO] Connected to vcsa5 version 7.0.3 build 21290409
2023-04-07T12:03:17.7230149+02:00 [INFO] Start processing vCenter/ESX vcsa5 ...
2023-04-07T12:03:17.7455432+02:00 [INFO] Collecting objects in vcsa5 ...
2023-04-07T12:03:18.5768093+02:00 [INFO] Building hashtables ...
2023-04-07T12:03:18.6054856+02:00 [INFO] Writing Vm Inventory files ...
2023-04-07T12:03:18.6974830+02:00 [INFO] Writing Esx Inventory files ...
2023-04-07T12:03:18.7010685+02:00 [INFO] Writing Datastore Inventory files ...
PS>$global:?
True


PowerShell transcript end
End time: 20230407120318


@rschitz
Copy link
Member

rschitz commented Apr 7, 2023

i'd need /var/log/sexigraf/ViPullStatistics.log but if that's ok send it to plot[at]sexigraf.fr non anonymized so i can check if it has something to do with the naming

@n0405
Copy link
Author

n0405 commented Apr 7, 2023

that log has a lot information. but I checked the log and find something interesting

there are a lot lines in log like this :
2023-04-07T13:30:41.4034645+02:00 [EROR] ESX esxi-3 fatstats metrics issue in cluster cluster2
2023-04-07T13:30:41.4037475+02:00 [EROR] Cannot index into a null array.
2023-04-07T13:30:41.4277225+02:00 [EROR] ESX esxi-2 fatstats metrics issue in cluster cluste2
2023-04-07T13:30:41.4280100+02:00 [EROR] Cannot index into a null array.
2023-04-07T13:30:41.4558160+02:00 [EROR] ESX esxi-1 fatstats metrics issue in cluster cluster2
2023-04-07T13:30:41.4561033+02:00 [EROR] Cannot index into a null array.

rschitz added a commit that referenced this issue Apr 7, 2023
@rschitz
Copy link
Member

rschitz commented Apr 7, 2023

it's really strange because that imply the vcenter didnt return data for the power metric where it usually return 0 at the very least.

can you try the last code i've commited to see if it correct your particular issue? (you just need to replace this file or replace the code yourself you prefer)

9a22f07

@n0405
Copy link
Author

n0405 commented Apr 11, 2023

Hi,
I replaced opt/sexigraf/ViPullStatistics.ps1 , but still same behavior , same problem , I will try to deploy new appliance and check if I have same issues.

@rschitz
Copy link
Member

rschitz commented Apr 11, 2023

no need to redeploy, it wont solve anything. let me try to isolate the problematic metric, i'll post another ps1 and let you know to test. thanks again for your support.

@n0405
Copy link
Author

n0405 commented Apr 11, 2023

Hi , I did redeploy , I try different account to connect to vcsa but still same behavior.
I really like sexygraf , it is so cool monitoring tool , thank you for your help .

rschitz added a commit that referenced this issue Apr 11, 2023
@rschitz
Copy link
Member

rschitz commented Apr 11, 2023

@n0405
Copy link
Author

n0405 commented Apr 11, 2023

you did it , thanks
Capture

@rschitz
Copy link
Member

rschitz commented Apr 11, 2023

great, can you check the logs to see what is skipped then?

@n0405
Copy link
Author

n0405 commented Apr 11, 2023

PowerShell transcript start
Start time: 20230411144003


Transcript started, output file is /var/log/sexigraf/ViPullStatistics.log
2023-04-11T14:40:03.5566862+02:00 [INFO] ViPullStatistics v0.9.1013
2023-04-11T14:40:03.5746523+02:00 [INFO] Importing PowerCli and Graphite PowerShell modules ...
2023-04-11T14:40:13.5948335+02:00 [INFO] Looking for another ViPullStatistics for vcsa3 ...
2023-04-11T14:40:13.6527616+02:00 [INFO] SessionToken found in SessionFile, attempting connection to vcsa3 ...
2023-04-11T14:40:21.2755753+02:00 [INFO] Connected to vCenter vcsa3 version 7.0.3 build 20990077
2023-04-11T14:40:21.2811070+02:00 [INFO] Start processing vCenter/ESX vcsa3 ...
2023-04-11T14:40:21.5455104+02:00 [INFO] Processing SessionManager & EventManager ...
2023-04-11T14:40:21.7850717+02:00 [INFO] Ses2023-04-11T14:40:21.8361195+02:00 [INFO] Connected to vCenter vcsa4.sb2023-04-11T14:40:22.3563605+02:00 [INFO] vCenter objects collect ...
2023-04-11T14:40:28.3451269+02:00 [INFO] Building objects tables ...
2023-04-11T14:40:28.5485787+02:00 [INFO] vCenter objects relationship discover ...
2023-04-11T14:40:28.6185689+02:00 [INFO] Performance metrics collection ...
2023-04-11T14:40:33.0058737+02:00 [INFO] All hosts multi metrics collected in 4.3808279 sec for vCenter vcsa3
2023-04-11T14:40:37.3013927+02:00 [INFO] All vms multi metrics collected in 4.2832702 sec for vCenter vcsa3
2023-04-11T14:40:37.3354838+02:00 [INFO] All Clusters multi metrics collected in 0.0244989 sec for vCenter vcsa3
2023-04-11T14:40:37.3492001+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 in datacenter vcsa3-DC
2023-04-11T14:40:37.3840443+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 hosts in datacenter vcsa3-DC
2023-04-11T14:40:38.1002444+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 vms in datacenter vcsa3-DC
2023-04-11T14:40:40.0592787+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 datastores in datacenter vcsa3-DC
2023-04-11T14:40:40.2888699+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster3 ...
2023-04-11T14:40:40.9892883+02:00 [INFO] Processing vCenter vcsa3 SessionList
2023-04-11T14:40:40.9899071+02:00 [INFO] Processing vCenter vcsa3 events
2023-04-11T14:40:41.4109935+02:00 [INFO] End processing vCenter vcsa3
PS>$global:?
True


PowerShell transcript end
End time: 20230411144041


s power metrics issue in cluster ne-sbb-corp
2023-04-11T14:40:35.6445067+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:40:35.6880135+02:00 [EROR] ESX vcsa2-host-7 fatstats power metric2023-04-11T14:40:35.7887929+02:00 [INFO] Processing vCenter vcsa5 cluster cluste5 in datacenter dc5
2023-04-11T14:40:35.8331581+02:00 [INFO] Processing vCenter vcsa5 cluster cluster5 hosts in datacenter dc5
2023-04-11T14:40:36.1227668+02:00 [INFO] Processing vCenter vcsa5 cluster cluster5 vms in datacenter dc5
2023-04-11T14:40:36.5713822+02:00 [INFO] Processing vCenter vcsa5 cluster cluster5 datastores in datacenter dc5
2023-04-11T14:40:36.7216658+02:00 [INFO] Start processing VsanPerfQuery in cluster cluster5 (v6.7+) ...
2023-04-11T14:40:37.4028289+02:00 [INFO] VsanPerfQueryPerf metrics collected in 0.614513 sec for vSAN Cluster cluster5 in vCenter vcsa5
2023-04-11T14:40:37.5955380+02:00 [INFO] Start processing SmartStatsSummary in cluster cluster5 (v6.7+) ...
2023-04-11T14:40:38.5549892+02:00 [INFO] Processing spaceUsageByObjectType in vSAN cluster cluster5 (v6.2+) ...
2023-04-11T14:40:39.2108135+02:00 [INFO] Start processing SyncingVsanObjectsSummary in cluster cluster5 (v6.7+) ...
2023-04-11T14:40:39.4912386+02:00 [INFO] Start processing VsanObjectIdentityAndHealth in cluster cluster5 ...
2023-04-11T14:40:39.7486563+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster5 ...
2023-04-11T14:40:40.1063789+02:00 [INFO] Processing vCenter vcsa5 SessionList
2023-04-11T14:40:40.1067238+02:00 [INFO] Processing vCenter vcsa5 events
2023-04-11T14:40:40.6068690+02:00 [INFO] vCenter vcsa5 no new event collected
2023-04-11T14:40:40.6462506+02:00 [INFO] End processing vCenter vcsa5
PS>$global:?
True


PowerShell transcript end
End time: 20230411144040


:40:38.0071994+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:40:38.0165994+02:00 [EROR] ESX vcsa2-host-a-2 fatstats power metrics issue in cluster cluster2-a
2023-04-11T14:40:38.0173462+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:40:38.0319907+02:00 [EROR] ESX vcsa2-host-a-1 fatstats power metrics issue in cluster cluster2-a
2023-04-11T14:40:38.0328188+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:40:38.0343399+02:00 [INFO] Processing vCenter vcsa2 cluster cluster2-a vms in datacenter cluster2-a
2023-04-11T14:40:38.2694251+02:00 [INFO] Processing vCenter vcsa2 cluster cluster2-a datastores in datacenter cluster2-a
2023-04-11T14:40:38.3407744+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster2-a ...
2023-04-11T14:40:38.7287045+02:00 [INFO] Processing vCenter vcsa2 SessionList
2023-04-11T14:40:38.7309164+02:00 [INFO] Processing vCenter vcsa2 events
2023-04-11T14:40:39.4173918+02:00 [INFO] End processing vCenter vcsa2
PS>$global:?
True


PowerShell transcript end
End time: 20230411144039


ts power metrics issue in cluster cluster1-b
2023-04-11T14:40:38.3398528+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:40:38.3521023+02:00 [EROR] ESX vcsa1-stan-1 fatstats power metrics issue in cluster cluster1-b
2023-04-11T14:40:38.3677628+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:40:38.3695612+02:00 [INFO] Processing vCenter vcsa1 cluster cluster1-b vms in datacenter DC1-1
2023-04-11T14:40:38.3862517+02:00 [INFO] Processing vCenter vcsa1 cluster cluster1-b datastores in datacenter DC1-1
2023-04-11T14:40:38.3892579+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster1-b ...
2023-04-11T14:40:38.4775307+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2in datacenter vcsa1-dc2
2023-04-11T14:40:38.5606945+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2datastores in datacenter vcsa1-dc2
2023-04-11T14:40:38.5771931+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2vms in datacenter vcsa1-dc2
2023-04-11T14:40:38.8173256+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-4 in datacenter vcsa1-dc2
2023-04-11T14:40:38.8255946+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-4 datastores in datacenter vcsa1-dc2
2023-04-11T14:40:38.8359163+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-4 vms in datacenter vcsa1-dc2
2023-04-11T14:40:39.0332154+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 in datacenter vcsa1-dc2
2023-04-11T14:40:39.0525701+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 datastores in datacenter vcsa1-dc2
2023-04-11T14:40:39.0539605+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 vms in datacenter vcsa1-dc2
2023-04-11T14:40:39.2658885+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-3 in datacenter vcsa1-dc2
2023-04-11T14:40:39.2785180+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-3 datastores in datacenter vcsa1-dc2
2023-04-11T14:40:39.2986742+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-3 vms in datacenter vcsa1-dc2
2023-04-11T14:40:39.5255462+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-5 in datacenter vcsa1-dc2
2023-04-11T14:40:39.5324834+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-5 datastores in datacenter vcsa1-dc2
2023-04-11T14:40:39.5689239+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-5 vms in datacenter vcsa1-dc2
2023-04-11T14:40:39.6572298+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-4 in datacenter cluster1
2023-04-11T14:40:39.6926153+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-4 datastores in datacenter cluster1
2023-04-11T14:40:39.7055659+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-4 vms in datacenter cluster1
2023-04-11T14:40:39.7939920+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 in datacenter vcsa1-dc2
2023-04-11T14:40:39.8502429+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 datastores in datacenter vcsa1-dc2
2023-04-11T14:40:39.8516456+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 vms in datacenter vcsa1-dc2
2023-04-11T14:40:40.0284094+02:00 [INFO] Processing vCenter vcsa1 SessionList
2023-04-11T14:40:40.0288248+02:00 [INFO] Processing vCenter vcsa1 events
2023-04-11T14:40:40.5641689+02:00 [INFO] End processing vCenter vcsa1
PS>$global:?
True


PowerShell transcript end
End time: 20230411144040



PowerShell transcript start
Start time: 20230411144503


Transcript started, output file is /var/log/sexigraf/ViPullStatistics.log
2023-04-11T14:45:03.8927948+02:00 [INFO] ViPullStatistics v0.9.1013
2023-04-11T14:45:03.9067543+02:00 [INFO] Importing PowerCli and Graphite PowerShell modules ...
2023-04-11T14:45:13.5281970+02:00 [INFO] Looking for another ViPullStatistics for vcsa2 ...
2023-04-11T14:45:13.5899056+02:00 [INFO] SessionToken found in SessionFile, attempting connection to vcsa2 ...
2023-04-11T14:45:21.3048590+02:00 [INFO] Connected to vCenter vcsa2 version 7.0.3 build 20990077
2023-04-11T14:45:21.3256717+02:00 [INFO] Start processing vCenter/ESX vcsa2 ...
2023-04-11T14:45:21.6799095+02:00 [INFO] Processing SessionManager & EventManager ...
2023-04-11T14:45:21.7776184+02:00 [INFO] Sessions.TerminateSession privilege not detected, SessionManager skipped ...
2023-04-11T14:45:22.3807848+02:00 [INFO] vCenter objects collect ...
2023-04-11T14:45:28.0274078+02:00 [INFO] Building objects tables ...
2023-04-11T14:45:28.1953868+02:00 [INFO] vCenter objects relationship discover ...
2023-04-11T14:45:28.2795449+02:00 [INFO] Performance metrics collection ...
2023-04-11T14:45:31.5758859+02:00 [INFO] All hosts multi metrics collected in 3.2941041 sec for vCenter vcsa2
2023-04-11T14:45:35.2292510+02:00 [INFO] All vms multi metrics collected in 3.6372419 sec for vCenter vcsa2
2023-04-11T14:45:35.2827938+02:00 [INFO] All Clusters multi metrics collected in 0.0413093 sec for vCenter vcsa2
2023-04-11T14:45:35.2977608+02:00 [INFO] Processing vCenter vcsa2 cluster cluster2-a in datacenter cluster2-a
2023-04-11T14:45:35.3495128+02:00 [INFO] Processing vCenter vcsa2 cluster cluster2-a hosts in datacenter cluster2-a
2023-04-11T14:45:35.5255994+02:00 [EROR] ESX vcsa2-host-a-3 fatstats power metrics issue in cluster cluster2-a
2023-04-11T14:45:35.5434654+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:45:35.6581296+022023-04-11T14:45:37.1730786+02:00 [INFO] All vms multi metrics collected in 4.0967861 sec for vCenter vcsa3
2023-04-11T14:45:37.2315304+02:00 [INFO] All Clusters multi metrics collected in 0.0415398 sec for vCenter vcsa3
2023-04-11T14:45:37.2497188+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 in datacenter tp-corp
2023-04-11T14:45:37.3094520+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 hosts in datacenter tp-corp
2023-04-11T14:45:38.2268187+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 vms in datacenter tp-corp
2023-04-11T14:45:39.7814455+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 datastores in datacenter tp-corp
2023-04-11T14:45:40.2447640+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster3 ...
2023-04-11T14:45:41.1012507+02:00 [INFO] Processing vCenter vcsa3 SessionList
2023-04-11T14:45:41.1114598+02:00 [INFO] Processing vCenter vcsa3 events
2023-04-11T14:45:41.6145647+02:00 [INFO] End processing vCenter vcsa3
PS>$global:?
True


PowerShell transcript end
End time: 20230411144541


e-sbb-corp
2023-04-11T14:45:37.4591315+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:45:37.4666530+02:00 [EROR] ESX vcsa2-host-7 fatstats power metrics issue in cluster cluster2
2023-04-11T14:45:37.4674622+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:45:37.4991874+02:00 [EROR] ESX vcsa2-host-a-8 fatstats power metrics issue in cluster cluster2
2023-04-11T14:45:37.6120881+02:00 [INFO] Start processing SmartStatsSummary in cluster pg (v6.7+) ...
2023-04-11T14:45:38.5522899+02:00 [INFO] Processing spaceUsageByObjectType in vSAN cluster cluster5 (v6.2+) ...
2023-04-11T14:45:39.0502316+02:00 [INFO] Start processing SyncingVsanObjectsSummary in cluster cluster5 (v6.7+) ...
2023-04-11T14:45:39.1668028+02:00 [INFO] Start processing VsanObjectIdentityAndHealth in cluster cluster5 ...
2023-04-11T14:45:39.4099958+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster pg ...
2023-04-11T14:45:40.0627331+02:00 [INFO] Processing vCenter vcsa5 SessionList
2023-04-11T14:45:40.0714379+02:00 [INFO] Processing vCenter vcsa5 events
2023-04-11T14:45:40.6597122+02:00 [INFO] vCenter vcsa5 no new event collected
2023-04-11T14:45:40.6795146+02:00 [INFO] End processing vCenter vcsa5
PS>$global:?
True


PowerShell transcript end
End time: 20230411144540


stores in datacenter vcsa2-DC
2023-04-11T14:45:38.3737809+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster ne-sbb-corp ...
2023-04-11T14:45:39.0044158+02:00 [INFO] Processing vCenter vcsa2 SessionList
2023-04-11T14:45:39.0047337+02:00 [INFO] Processing vCenter vcsa2 events
2023-04-11T14:45:39.5459690+02:00 [INFO] End processing vCenter vcsa2
PS>$global:?
True


PowerShell transcript end
End time: 20230411144539


co-cloud
2023-04-11T14:45:37.2537235+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster1 ...
2023-04-11T14:45:37.9074921+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 in datacenter vcsa1-dc2
2023-04-11T14:45:37.9968641+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 datastores in datacenter vcsa1-dc2
2023-04-11T14:45:38.0126625+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 vms in datacenter vcsa1-dc2
2023-04-11T14:45:38.1937915+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-3 in datacenter vcsa1-dc2
2023-04-11T14:45:38.2084778+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-3 datastores in datacenter vcsa1-dc2
2023-04-11T14:45:38.2169503+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-3 vms in datacenter vcsa1-dc2
2023-04-11T14:45:38.3502243+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 in datacenter vcsa1-dc2
2023-04-11T14:45:38.3640137+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 datastores in datacenter vcsa1-dc2
2023-04-11T14:45:38.3935179+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 vms in datacenter vcsa1-dc2
2023-04-11T14:45:38.6021728+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-5 in datacenter vcsa1-dc2
2023-04-11T14:45:38.6186462+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-5 datastores in datacenter vcsa1-dc2
2023-04-11T14:45:38.6459418+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-5 vms in datacenter vcsa1-dc2
2023-04-11T14:45:38.7963450+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-4 in datacenter cluster1
2023-04-11T14:45:38.8437892+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-4 datastores in datacenter cluster1
2023-04-11T14:45:38.8795944+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-4 vms in datacenter cluster1
2023-04-11T14:45:38.9938313+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2in datacenter vcsa1-dc2
2023-04-11T14:45:38.9982865+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2datastores in datacenter vcsa1-dc2
2023-04-11T14:45:39.0093398+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2vms in datacenter vcsa1-dc2
2023-04-11T14:45:39.1781637+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-4 in datacenter vcsa1-dc2
2023-04-11T14:45:39.1877306+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-4 datastores in datacenter vcsa1-dc2
2023-04-11T14:45:39.2075350+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-4 vms in datacenter vcsa1-dc2
2023-04-11T14:45:39.2636060+02:00 [INFO] Processing vCenter vcsa1 SessionList
2023-04-11T14:45:39.2639327+02:00 [INFO] Processing vCenter vcsa1 events
2023-04-11T14:45:39.9659430+02:00 [INFO] End processing vCenter vcsa1
PS>$global:?
True


PowerShell transcript end
End time: 20230411144539



PowerShell transcript start
Start time: 20230411145004


Transcript started, output file is /var/log/sexigraf/ViPullStatistics.log
2023-04-11T14:50:04.6953674+02:00 [INFO] ViPullStatistics v0.9.1013
2023-04-11T14:50:04.7101622+02:00 [INFO] Importing PowerCli and Graphite PowerShell modules ...
2023-04-11T14:50:15.0606945+02:00 [INFO] Looking for another ViPullStatistics for vcsa2 ...
2023-04-11T14:50:15.1102702+02:00 [INFO] SessionToken found in SessionFile, attempting connection to vcsa2 ...
2023-04-11T14:50:23.2675937+02:00 [INFO] Connected to vCenter vcsa2 version 7.0.3 build 20990077
2023-04-11T14:50:23.2857271+02:00 [INFO] Start processing vCenter/ESX vcsa2 ...
2023-04-11T14:50:23.6618611+02:00 [INFO] Processing SessionManager & EventManager ...
2023-04-11T14:50:23.9817194+02:00 [INFO] Sessions.TerminateSession privilege not detected, SessionManager skipped ...
2023-04-11T14:50:24.6760392+02:00 [INFO] vCenter objects collect ...
2023-04-11T14:50:30.5444301+02:00 [INFO] Building objects tables ...
2023-04-11T14:50:30.7579491+02:00 [INFO] vCenter objects relationship discover ...
2023-04-11T14:50:30.8694203+02:00 [INFO] Performance metrics collection ...
2023-04-11T14:50:34.0084917+02:00 [INFO] All hosts multi metrics collected in 3.1367397 sec for vCenter vcsa2
2023-04-11T14:50:37.3284842+02:00 [INFO] All vms multi metrics collected in 3.304015 sec for vCenter vcsa2
2023-04-11T14:50:37.3605979+02:00 [INFO] All Clusters multi metrics collected in 0.0258329 sec for vCenter vcsa2
2023-04-11T14:50:37.3826293+02:00 [INFO] Processing vCenter vcsa2 cluster cluster2-a in datacenter cluster2-a
2023-04-11T14:50:37.4381790+02:00 [INFO] Processing vCenter vcsa2 cluster cluster2-a hosts in datacenter cluster2-a
2023-04-11T14:50:37.6245644+02:00 [EROR] ESX vcsa2-host-a-3 fatstats power metrics issue in cluster cluster2-a
2023-04-11T14:50:37.6248959+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:50:37.6968313+02:00 [EROR] ESX vcsa2-host-a-2 fatstats power metrics issue in cluster cluster2-a
2023-04-11T14:50:37.6971487+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:50:37.8213001+02:00 [EROR] ESX vcsa2-hos-2-2023-04-11T14:50:37.9883641+02:00 [INFO] Processing vCenter vcsa5 cluster pg in datacenter dc5
2023-04-11T14:50:38.0360515+02:00 [INFO] Processing vCenter vcsa5 cluster pg hosts in datacenter dc5
2023-04-11T2023-04-11T14:50:38.9605737+02:00 [INFO] Processing vCenter vcsa2 cluster cluster2-a datastores in datacenter cluster2-a
2023-04-11T14:50:39.1828626+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster2-a ...
2023-04-11T14:50:39.72023-04-11T14:50:40.5650879+02:00 [INFO] All vms multi metrics collected in 4.3713598 sec for vCenter vcsa3
2023-04-11T14:50:40.6080164+02:00 [INFO] All Clusters multi metrics collected in 0.0301922 sec for vCenter vcsa3
2023-04-11T14:50:40.6173702+02:00 [INFO] Processing vCent2023-04-11T14:50:40.9243308+02:00 [INFO] Processing spaceUsageByObjectType in vSAN cluster cluster5(v6.2+) ...
2023-04-11T14:50:41.5136745+02:00 [INFO] Start processing SyncingVsanObjectsSummary in cluster cluster5 (v6.7+) ...
2023-04-11T14:50:41.6295165+02:00 [INFO] Start processing VsanObjectIdentityAndHealth in cluster cluster5 ...2023-04-11T14:50:42.5553191+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 datastores in datacenter DC3
2023-04-11T14:50:42.7141095+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster3 ...
2023-04-11T14:50:43.3420534+02:00 [INFO] Processing vCenter vcsa3 SessionList
2023-04-11T14:50:43.3428748+02:00 [INFO] Processing vCenter vcsa3 events
2023-04-11T14:50:43.7719653+02:00 [INFO] End processing vCenter vcsa3
PS>$global:?
True


PowerShell transcript end
End time: 20230411145043


9.9610592+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:50:40.0004925+02:00 [EROR] ESX vcsa2-host-a-6 fatstats power metrics issue in cluster ne-sbb-corp
2023-04-11T14:50:40.0195990+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:50:40.0497429+02:00 [EROR] ESX vcsa2/hos-5fatstats power metrics issue in cluster ne-sbb-corp
2023-04-11T14:50:40.0501370+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:50:40.0514542+02:00 [INFO] Processing vCenter vcsa2 cluster cluste2vms in datacenter sbb-aok
2023-04-11T14:50:40.5366272+02:00 [INFO] Processing vCenter vcsa2 cluster cluster2 datastores in datacenter sbb-aok
2023-04-11T14:50:40.6887994+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster ne-sbb-corp ...
2023-04-11T14:50:41.2192972+02:00 [INFO] Processing vCenter vcsa2 SessionList
2023-04-11T14:50:41.2198560+02:00 [INFO] Processing vCenter vcsa2 events
2023-04-11T14:50:41.7688563+02:00 [INFO] End processing vCenter vcsa2
PS>$global:?
True


PowerShell transcript end
End time: 20230411145041


metrics issue in cluster cluster1
2023-04-11T14:50:37.9995221+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:50:38.0130043+02:00 [INFO] Processing vCenter vcsa1 cluster cluster1 vms in datacenter cluster1
2023-04-11T14:50:38.9492428+02:00 [EROR] VM test-tt storage commit metric issue in cluster cluster1
2023-04-11T14:50:38.9599231+02:00 [EROR] Exception calling "Add" with "2" argument(s): "Item has already been added. Key in dictionary: 'vmw.vcsa1.cluster1.cluster1.vm.test-tt.storage.committed' Key being added: 'vmw.vcsa1.cluster1.cluster1.vm.ttv-cro_test.storage.committed'"
2023-04-11T14:50:38.9993691+02:00 [INFO] Processing vCenter vcsa1 cluster cluster1 datastores in datacenter cluster1
2023-04-11T14:50:39.1909313+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster1 ...
2023-04-11T14:50:39.6612699+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2in datacenter vcsa1-dc2
2023-04-11T14:50:39.7300836+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2datastores in datacenter vcsa1-dc2
2023-04-11T14:50:39.7435157+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2vms in datacenter vcsa1-dc2
2023-04-11T14:50:39.9646458+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-4 in datacenter vcsa1-dc2
2023-04-11T14:50:39.9732657+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-4 datastores in datacenter vcsa1-dc2
2023-04-11T14:50:39.9917245+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-4 vms in datacenter vcsa1-dc2
2023-04-11T14:50:40.1789940+02:00 [INFO] Processing vCenter vcsa1 standalone host tp-esxi-perf-5 in datacenter vcsa1-dc2
2023-04-11T14:50:40.2117065+02:00 [INFO] Processing vCenter vcsa1 standalone host tp-esxi-perf-5 datastores in datacenter vcsa1-dc2
2023-04-11T14:50:40.2537953+02:00 [INFO] Processing vCenter vcsa1 standalone host tp-esxi-perf-5 vms in datacenter vcsa1-dc2
2023-04-11T14:50:40.4650408+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 in datacenter vcsa1-dc2
2023-04-11T14:50:40.4784193+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 datastores in datacenter vcsa1-dc2
2023-04-11T14:50:40.5234235+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 vms in datacenter vcsa1-dc2
2023-04-11T14:50:40.6518131+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-3 in datacenter vcsa1-dc2
2023-04-11T14:50:40.6676848+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-3 datastores in datacenter vcsa1-dc2
2023-04-11T14:50:40.7134191+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-3 vms in datacenter vcsa1-dc2
2023-04-11T14:50:40.8468554+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 in datacenter vcsa1-dc2
2023-04-11T14:50:40.8781794+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 datastores in datacenter vcsa1-dc2
2023-04-11T14:50:40.8965086+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 vms in datacenter vcsa1-dc2
2023-04-11T14:50:41.0896590+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-4 in datacenter cluster1
2023-04-11T14:50:41.1317630+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-4 datastores in datacenter cluster1
2023-04-11T14:50:41.1419780+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-4 vms in datacenter cluster1
2023-04-11T14:50:41.1964319+02:00 [INFO] Processing vCenter vcsa1 SessionList
2023-04-11T14:50:41.2028177+02:00 [INFO] Processing vCenter vcsa1 events
2023-04-11T14:50:41.8529367+02:00 [INFO] End processing vCenter vcsa1
PS>$global:?
True


PowerShell transcript end
End time: 20230411145041



PowerShell transcript start
Start time: 20230411145503


Transcript started, output file is /var/log/sexigraf/ViPullStatistics.log
2023-04-11T14:55:03.4392581+02:00 [INFO] ViPullStatistics v0.9.1013
2023-04-11T14:55:03.4581745+02:00 [INFO] Importing PowerCli and Graphite PowerShell modules ...
2023-04-11T14:55:13.0649676+02:00 [INFO] Looking for another ViPullStatistics for vcsa3 ...
2023-04-11T14:55:13.1481975+02:00 [INFO] SessionToken found in SessionFile, attempting connection to vcsa3 ...
2023-04-11T14:55:20.9773303+02:00 [INFO] Connected to vCenter vcsa3 version 7.0.3 build 20990077
2023-04-11T14:55:20.9940513+02:00 [INFO] Start processing vCenter/ESX vcsa3 ...
2023-04-11T14:55:21.2800388+02:00 [INFO] Processing SessionManager & EventManager ...
2023-04-11T14:55:21.4613752+02:00 [INFO] Sessions.TerminateSession privilege not detected, SessionManager skipped ...
2023-04-11T14:55:21.9718028+02:00 [INFO] vCenter objects collect ...
2023-04-11T14:55:28.2002035+02:00 [INFO] Building objects tables ...
2023-04-11T14:55:28.4332402+02:00 [INFO] vCenter objects relationship discover ...
2023-04-11T14:55:28.5256524+02:00 [INFO] Performance metrics collection ...
2023-04-11T14:55:32.7980409+02:00 [INFO] All hosts multi metrics collected in 4.2647596 sec for vCenter vcsa3
2023-04-11T14:55:37.5098540+02:00 [INFO] All vms multi metrics collected in 4.7009976 sec for vCenter vcsa3
2023-04-11T14:55:37.5649850+02:00 [INFO] All Clusters multi metrics collected in 0.0506081 sec for vCenter vcsa3
2023-04-11T14:55:37.5836695+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 in datacenter vcsa3-DC
2023-04-11T14:55:37.6135748+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 hosts in datacenter vcsa3-DC
2023-04-11T14:55:38.2426972+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 vms in datacenter vcsa3-DC
2023-04-11T14:55:39.9232684+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 datastores in datacenter vcsa3-DC
2023-04-11T14:55:40.2913222+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster3 ...
2023-04-11T14:55:41.0625353+02:00 [INFO] Processing vCenter vcsa3 SessionList
2023-04-11T14:55:41.0631139+02:00 [INFO] Processing vCenter vcsa3 events
2023-04-11T14:55:41.5849765+02:00 [INFO] End processing vCenter vcsa3
PS>$global:?
True


PowerShell transcript end
End time: 20230411145541


23-04-11T14:55:36.6991534+02:00 [INFO] Processing vCenter vcsa5 cluster pg datastores in datacenter dc5
2023-04-11T14:55:36.8502474+02:00 [INFO] Start processing VsanPerfQuery in cluster cluster5 (v6.7+) ...
2023-04-11T14:55:37.5652407+02:00 [INFO] VsanPerfQueryPerf metrics collected in 0.6331279 sec for vSAN Cluster pg in vCenter vcsa5
2023-04-11T14:55:37.7648306+02:00 [INFO] Start processing SmartStatsSummary in cluster cluster5 (v6.7+) ...
2023-04-11T14:55:38.9083022+02:00 [INFO] Processing spaceUsageByObjectType in vSAN cluster cluster5 (v6.2+) ...
2023-04-11T14:55:39.5694520+02:00 [INFO] Start processing SyncingVsanObjectsSummary in cluster cluster5 (v6.7+) ...
2023-04-11T14:55:39.6887242+02:00 [INFO] Start processing VsanObjectIdentityAndHealth in cluster cluster5 ...
2023-04-11T14:55:39.9714586+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster pg ...
2023-04-11T14:55:40.4772003+02:00 [INFO] Processing vCenter vcsa5 SessionList
2023-04-11T14:55:40.4775545+02:00 [INFO] Processing vCenter vcsa5 events
2023-04-11T14:55:40.9382455+02:00 [INFO] vCenter vcsa5 no new event collected
2023-04-11T14:55:40.9488784+02:00 [INFO] End processing vCenter vcsa5
PS>$global:?
True


PowerShell transcript end
End time: 20230411145541


s power metrics issue in cluster ne-sbb-corp
2023-04-11T14:55:37.6511124+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:55:37.6582949+02:00 [EROR] ESX vcsa2-host-a-8 fatstats power metrics issue in cluster ne-sbb-corp
2023-04-11T14:55:37.6963127+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:55:37.7336635+02:00 [EROR] ESX vcsa2-host-a-1 fatstats power metrics issue in cluster ne-sbb-corp
2023-04-11T14:55:37.7673930+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:55:37.8092562+02:00 [EROR] ESX vcsa-host/-3 fatstats power metrics issue in cluster ne-sbb-corp
2023-04-11T14:55:37.8234614+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:55:37.8620649+02:00 [EROR] ESX vcsa2-host-a-6 fatstats power metrics issue in cluster ne-sbb-corp
2023-04-11T14:55:37.8677370+02:00 [EROR] Ca2023-04-11T14:55:37.9482510+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster1 ...
2023-04-11T14:55:38.6233155+02:00 [INFO] Processing vCenter vcsa1 cluster cluster1-b in datacenter DC1-1
2023-04-11T14:55:38.6238313+02:00 [INFO] Processing vCenter vcsa1 cluster cluster1-b hosts in datacenter DC1-1
2023-04-11T14:55:38.6424000+02:00 [EROR] ESX vcsa1-stan-2 fatstats power metrics issue in cluster cluster1-b
2023-04-11T14:55:38.6427271+02:00 [EROR] Cannot index into a null array.
2023-04-11T14:55:38.6512774+02:00 [EROR] ESX cona2023-04-11T14:55:39.0926047+02:00 [INFO] Processing vCenter vcsa2 SessionList
2023-04-11T14:55:39.0933533+02:00 [INFO] Processing vCenter vcsa2 events
2023-04-11T14:55:39.7993519+02:00 [INFO] End processing vCenter vcsa2
PS>$global:?
True


PowerShell transcript end
End time: 20230411145539


nter DC1-1
2023-04-11T14:55:38.6807797+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster1-b ...
2023-04-11T14:55:38.7721775+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 in datacenter vcsa1-dc2
2023-04-11T14:55:38.7971615+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 datastores in datacenter vcsa1-dc2
2023-04-11T14:55:38.8048464+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 vms in datacenter vcsa1-dc2
2023-04-11T14:55:38.9940314+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-4 in datacenter cluster1
2023-04-11T14:55:39.0375289+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-4 datastores in datacenter cluster1
2023-04-11T14:55:39.1176854+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-4 vms in datacenter cluster1
2023-04-11T14:55:39.4642484+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-4 in datacenter vcsa1-dc2
2023-04-11T14:55:39.4772609+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-4 datastores in datacenter vcsa1-dc2
2023-04-11T14:55:39.5172178+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-4 vms in datacenter vcsa1-dc2
2023-04-11T14:55:39.6743084+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-3 in datacenter vcsa1-dc2
2023-04-11T14:55:39.6967276+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-3 datastores in datacenter vcsa1-dc2
2023-04-11T14:55:39.6980776+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-3 vms in datacenter vcsa1-dc2
2023-04-11T14:55:39.8379767+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2in datacenter vcsa1-dc2
2023-04-11T14:55:39.8716596+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2datastores in datacenter vcsa1-dc2
2023-04-11T14:55:39.8820952+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2vms in datacenter vcsa1-dc2
2023-04-11T14:55:40.0144456+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 in datacenter vcsa1-dc2
2023-04-11T14:55:40.0353343+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 datastores in datacenter vcsa1-dc2
2023-04-11T14:55:40.0403272+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 vms in datacenter vcsa1-dc2
2023-04-11T14:55:40.1995063+02:00 [INFO] Processing vCenter vcsa1 SessionList
2023-04-11T14:55:40.1998151+02:00 [INFO] Processing vCenter vcsa1 events
2023-04-11T14:55:40.8692200+02:00 [INFO] End processing vCenter vcsa1
PS>$global:?
True


PowerShell transcript end
End time: 20230411145540



PowerShell transcript start
Start time: 20230411150004


Transcript started, output file is /var/log/sexigraf/ViPullStatistics.log
2023-04-11T15:00:04.1889315+02:00 [INFO] ViPullStatistics v0.9.1013
2023-04-11T15:00:04.2133766+02:00 [INFO] Importing PowerCli and Graphite PowerShell modules ...
2023-04-11T15:00:13.8978683+02:00 [INFO] Looking for another ViPullStatistics for vcsa4 ...
2023-04-11T15:00:13.9635673+02:00 [INFO] SessionToken found in SessionFile, attempting connection to vcsa4 ...
2023-04-11T15:00:21.7601880+02:00 [INFO] Connected to vCenter vcsa4 version 7.0.3 build 21477706
2023-04-11T15:00:21.7793893+02:00 [INFO] Start processing vCenter/ESX vcsa4l ...
2023-04-11T15:00:22.07182023-04-11T15:00:22.5467277+02:00 [INFO] Connected to vCenter 192.168.3.78 version 7.0.3 build 20990077
2023-04-11T15:00:22.5587236+02:00 [INFO] Start processing vCenter/ESX 192.162023-04-11T2023-04-11T15:00:22.8477283+02:00 [INFO] Processing Sessio2023-04-11T15:00:27.2048963+02:00 [INFO] Building objects tables ...
2023-04-11T15:00:27.3271019+02:00 [INFO] vCenter objects relationship discover ...
2023-04-11T15:00:27.4813293+02:00 [INFO] Performance metrics co2023-04-11T15:00:29.5897179+02:00 [INFO] Building objects tables ...
2023-04-11T15:00:29.7587550+02:00 [INFO] vCenter objects relationship discover ...
2023-04-11T15:00:29.8653669+02:00 [INFO] Performance metrics collection ...
2023-04-11T15:00:33.1653698+02:00 [INFO] All hosts multi metrics collected in 3.2898407 sec for vCenter vcsa1
2023-04-11T15:00:35.9075536+02:00 [INFO] All vms multi metrics collected in 2.7183171 sec for vCenter vcsa1
2023-04-11T15:00:35.9631244+02:00 [INFO] All Clusters multi metrics collected in 0.0411886 sec for vCenter vcsa1
2023-04-11T15:00:35.9893440+02:00 [INFO] Processing vCenter vcsa1 cluster cluster1-b in datacenter DC1-1
2023-04-11T15:00:36.0335105+02:00 [INFO] Processing vCenter vcsa1 cluster cluster1-b hosts in datacenter DC1-1
2023-04-11T15:00:36.1821919+02:00 [EROR] ESX vcsa1-stan-2 fatstats power metrics issue in cluster cluster1-b
2023-04-11T15:00:36.1828097+02:00 [EROR] Cannot index into a null array.
2023-04-11T15:00:36.2124485+02:00 [EROR] ESX vcsa1-stan-1 fatstats power metrics issue in cluster cluster1-b
2023-04-11T15:00:36.2211768+02:00 [EROR] Cannot index into a null array.
2023-04-11T15:00:36.2278753+02:00 [INFO] Processing vCenter vcsa1 cl2023-04-11T15:00:37.7441224+02:00 [INFO] All vms multi metrics collected in 4.3245646 sec for vCenter vcsa3
2023-04-11T15:00:37.7850102+02:00 [INFO] All Clusters multi metrics collected in 0.0325467 sec for vCenter vcsa3
2023-04-11T15:00:37.8027980+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 in datacenter vcsa3-DC
2023-04-11T15:00:37.8672898+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 hosts in datacenter vcsa3-DC
2023-04-11T15:00:38.5174505+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 vms in datacenter vcsa3-DC
2023-04-11T15:00:40.3124852+02:00 [INFO] Processing vCenter vcsa3 cluster cluster3 datastores in datacenter vcsa3-DC
2023-04-11T15:00:40.6905433+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster3 ...
2023-04-11T15:00:41.6325547+02:00 [INFO] Processing vCenter vcsa3 SessionList
2023-04-11T15:00:41.6329016+02:00 [INFO] Processing vCenter vcsa3 events
2023-04-11T15:00:42.0762645+02:00 [INFO] End processing vCenter vcsa3
PS>$global:?
True


PowerShell transcript end
End time: 20230411150042


ts, vms and datastores metrics to Graphite for cluster cluster5 ...
2023-04-11T15:00:40.6765144+02:00 [INFO] Processing vCenter vcsa5 SessionList
2023-04-11T15:00:40.6856885+02:00 [INFO] Processing vCenter vcsa5 events
2023-04-11T15:00:41.1789140+02:00 [INFO] vCenter vcsa5 no new event collected
2023-04-11T15:00:41.2014106+02:00 [INFO] End processing vCenter vcsa5
PS>$global:?
True


PowerShell transcript end
End time: 20230411150041


o a null array.
2023-04-11T15:00:37.3698353+02:00 [EROR] ESX vcsa1-host-a-7 fatstats power metrics issue in cluster cluster1
2023-04-11T15:00:37.3912422+02:00 [EROR] Cannot index into a null array.
2023-04-11T15:00:37.4003683+02:00 [INFO] Processing vCenter vcsa1 cluster cluster1 vms in datacenter cluster1
2023-04-11T15:00:38.2300950+02:00 [EROR] VM test-tt storage commit metric issue in cluster cluster1
2023-04-11T15:00:38.2304244+02:00 [EROR] Exception calling "Add" with "2" argument(s): "Item has already been added. Key in dic2023-04-11T15:00:38.2836927+02:00 [INFO] Processing vCenter vcsa2 cluster cluster2 datastores in datacenter vcsa2-DC2
2023-04-11T15:00:38.4130150+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster2 ...
2023-04-11T15:00:39.1228953+02:00 [INFO] Processing vCenter vcsa2 cluster cluster2-a in datacenter cluster2-a
2023-04-11T15:00:39.1312076+02:00 [INFO] Processing vCenter vcsa2 cluster cluster2-a hosts in datacenter cluster2-a
2023-04-11T15:00:39.1684800+02:00 [EROR] ESX vcsa2-host-a-3 fatstats power metrics issue in cluster cluster2-a
2023-04-11T15:00:39.1693528+02:00 [EROR] Cannot index into a null array.
2023-04-11T15:00:39.1994865+02:00 [EROR] ESX vcsa2-host-a-2 fatstats power metrics issue in cluster cluster2-a
2023-04-11T15:00:39.2002992023-04-11T15:00:39.2087571+02:00 [INFO] Proces2023-04-11T15:00:39.2410268+02:00 [EROR] ESX vcsa2-host-a-1 fatstats power metrics issue in cluster cluster2-a
2023-04-11T15:00:39.2522516+02:00 [EROR] Cannot index into a null array.
2023-04-11T15:00:39.2563098+02:00 [INFO] Processing vCenter vcsa2 cluster cluster2-a vms in datacenter cluster2-a
2023-04-11T15:00:39.4985944+02:00 [INFO] Processing vCenter vcsa2 cluster cluster2-a datastores in datacenter cluster2-a
2023-04-11T15:00:39.5499569+02:00 [INFO] Sending cluster, hosts, vms and datastores metrics to Graphite for cluster cluster2-a ...
2023-04-11T15:00:39.9726683+02:00 [INFO] Processing vCenter vcsa2 SessionList
2023-04-11T15:00:39.9757077+02:00 [INFO] Processing vCenter vcsa2 events
2023-04-11T15:00:40.6848307+02:00 [INFO] End processing vCenter vcsa2
PS>$global:?
True


PowerShell transcript end
End time: 20230411150040


_168_3_78 standalone host vcsa1-host-4 datastores in datacenter cluster1
2023-04-11T15:00:39.7504168+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-4 vms in datacenter cluster1
2023-04-11T15:00:39.9459491+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 in datacenter vcsa1-dc2
2023-04-11T15:00:39.9750064+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 datastores in datacenter vcsa1-dc2
2023-04-11T15:00:39.9969891+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-1 vms in datacenter vcsa1-dc2
2023-04-11T15:00:40.2176158+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2in datacenter vcsa1-dc2
2023-04-11T15:00:40.2368647+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2datastores in datacenter vcsa1-dc2
2023-04-11T15:00:40.2484152+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-2vms in datacenter vcsa1-dc2
2023-04-11T15:00:40.3377037+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 in datacenter vcsa1-dc2
2023-04-11T15:00:40.3443802+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 datastores in datacenter vcsa1-dc2
2023-04-11T15:00:40.3454918+02:00 [INFO] Processing vCenter vcsa1 standalone host vcsa1-host-b-6 vms in datacenter vcsa1-dc2
2023-04-11T15:00:40.4169710+02:00 [INFO] Processing vCenter vcsa1 SessionList
2023-04-11T15:00:40.4298807+02:00 [INFO] Processing vCenter vcsa1 events
2023-04-11T15:00:41.0778587+02:00 [INFO] End processing vCenter vcsa1
PS>$global:?

@rschitz
Copy link
Member

rschitz commented Apr 11, 2023

ok so it's the power metric, can you check in the vcenter see what you got on the power usage metric please?
image

@rschitz
Copy link
Member

rschitz commented Apr 11, 2023

this chart
image

@n0405
Copy link
Author

n0405 commented Apr 12, 2023

Hi ,
vCSA has power metric , I check and it has for all hosts
image

@rschitz
Copy link
Member

rschitz commented Apr 12, 2023

really strange, may i ask you to run a small manual test so i can understand what's going on and fix maybe?

@n0405
Copy link
Author

n0405 commented Apr 12, 2023

of course , what test were you referring to?

@rschitz
Copy link
Member

rschitz commented May 23, 2023

Hi, sorry for the delay.
I your /etc/cron.d/vi_[vcenter_fqdn] file you'll find something like
*/5 * * * * root /usr/bin/pwsh -NonInteractive -NoProfile -f /opt/sexigraf/ViPullStatistics.ps1 -credstore /mnt/wfs/inventory/vipscredentials.xml -server vcenter.vmware.com -sessionfile /tmp/vcenter_vmware_com.key >/dev/null 2>&1
In a ssh shell, run
pwsh
then
. /opt/sexigraf/ViPullStatistics.ps1 -credstore /mnt/wfs/inventory/vipscredentials.xml -server vcenter.vmware.com -sessionfile /tmp/vcenter_vmware_com.key
(replace with your vcenter fqdn)
when it's done, type this:
$HostMultiStats[$PerfCounterTable["power.power.average"]].values
and screenshot the results please

@rschitz rschitz added this to the 0.99j - St. Olga milestone May 23, 2023
@rschitz rschitz added the bug label May 23, 2023
@n0405
Copy link
Author

n0405 commented Jun 5, 2023

image

@rschitz
Copy link
Member

rschitz commented Jun 5, 2023

Great, thanks.
And what about this command $PerfCounterTable["power.power.average"] instead at the end please?

@n0405
Copy link
Author

n0405 commented Jun 5, 2023

image

@rschitz
Copy link
Member

rschitz commented Sep 18, 2023

seems to have been fixed in a later version of vCenter, @Nzaca do you have a chance to upgrade your setup?

@rschitz
Copy link
Member

rschitz commented Sep 27, 2023

missing power metrics solved in vcenter 7.0.3 21930508

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants