Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Frequently (every 1 minute) VsanPullStatistics from <vcenter> is already running! at /root/VsanPullStatistics.pl line 76 #174

Closed
eschek87 opened this issue Nov 16, 2018 · 21 comments
Assignees

Comments

@eschek87
Copy link

Every one minute I see the following in the /var/log/sexigraf/VsanDisksPullStatistics.log logfile:
/usr/local/share/perl/5.20.2/Log/Log4perl/Logger.pm line906 - DIE [ERROR] VsanPullStatistics from is already running! at /root/VsanPullStatistics.pl line 76

Therefore the vsan dashboards are empty when I select "last 1 hour". If I select "last 3 hours" I can see graphs.

Regards,
Stephan

@rschitz
Copy link
Member

rschitz commented Nov 16, 2018

how many vsan clusters and hosts per cluster you got?

@eschek87
Copy link
Author

eschek87 commented Nov 16, 2018

We have 4 all flash cluster:
cluster 1: 6 hosts
cluster 2: 8 hosts
cluster 3: 14 hosts
cluster 4: 16 hosts

And one hybrid cluster with 12 hosts. All as stretched clusters.

@rschitz
Copy link
Member

rschitz commented Nov 16, 2018

check the Pull Exec Time dashboard and give us the time the vi and vsan pull take please

@eschek87
Copy link
Author

2018-11-16 14_03_33-grafana - sexigraf pull exec time

@rschitz
Copy link
Member

rschitz commented Nov 16, 2018

ok it takes too long to complete. we're currently improving a lot of things but it shouldn't be that long.
What's the cpu spec of the appliance (vcpu and physical cpu underneath) ?

@eschek87
Copy link
Author

2 vCPUs (1 socket) and the host where it runs has 36 CPUs x 2.29 GHz (2 sockets, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz)
2018-11-16 14_19_43-vsphere web client

@rschitz
Copy link
Member

rschitz commented Nov 17, 2018

should be enough, let me get back to you when i'm done with the testing so you can try a new version of the puller script.

@rschitz
Copy link
Member

rschitz commented Feb 19, 2019

could you try this version of the vsan script https://github.com/sexibytes/sexigraf/blob/dev/root/VsanPullStatistics.pl

@rschitz
Copy link
Member

rschitz commented Feb 19, 2019

try also the new Vi script to see if the completion time is also better https://github.com/sexibytes/sexigraf/blob/dev/root/ViPullStatistics.pl

@eschek87
Copy link
Author

Pull Exec Time now looks better for vi statistics, but for vSAN it's similar. But I still have missing graphs in the "last 1 hour" view compared to the "last 3 hours view".
2019-02-21 10_21_05-grafana - vmware multi vsan monitor
2019-02-21 10_20_30-grafana - vmware multi vsan monitor
2019-02-21 10_18_56-grafana - sexigraf pull exec time

@rschitz
Copy link
Member

rschitz commented Feb 21, 2019

I found another way to increase processing speed, i'll get back to you ASAP. Thanks for your feedback!

@rschitz
Copy link
Member

rschitz commented Feb 22, 2019

@rschitz
Copy link
Member

rschitz commented Feb 22, 2019

Could you also send us a log extract from [INFO] Start processing vCenter to [INFO] End processing vCenter so we can have a better understanding of what is taking time to process please? send it to plot[at]sexigraf.fr if you have any security concern

@eschek87
Copy link
Author

eschek87 commented Feb 25, 2019

Hi,
I tried the new version and send you an email with the logfile. From the log I can see that it needs about 2 seconds per host and we have 56 vSAN hosts.

Regards,
Stephan

@rschitz
Copy link
Member

rschitz commented Feb 25, 2019

2sec is way too much, i'll have a close look into the logs

@rschitz rschitz self-assigned this Sep 27, 2020
@rschitz rschitz added this to the 0.99e - Highway 17 milestone Sep 27, 2020
@phroenips
Copy link

I'm having a similar issue on 0.99g

I only have a single two node vSAN cluster (ROBO, with remote witness appliance), and my vsan pull times are generally around 40 seconds. It exceeds 1 minute fairly regularly.

SexiGraf appliance has 2 vCPU and 8GB RAM; each physical host is dual socket Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz. I'll email the logs to plot[at]sexigraf.fr for review

Thanks!
Scott

@rschitz
Copy link
Member

rschitz commented Feb 6, 2021

Hi,
I tried the new version and send you an email with the logfile. From the log I can see that it needs about 2 seconds per host and we have 56 vSAN hosts.

Regards,
Stephan

I finally made some significant improvements with powershell parallel processing, would you be interested by some testing?

@matteowiz
Copy link

Hi,

I'm experiencing same problem after updating sexigraf from f to g version, we have 20 servers in 3 clusters and with previous version it takes an average of 30s to pull data. With new version we are over one minute and vsan charts are not displaying at all. Can I try new version you mentioned?

@rschitz
Copy link
Member

rschitz commented Mar 22, 2021

sure, reach me at plot[at]sexigraf.fr and i'll send you a link of the latest "beta" ;)

@hekyl
Copy link

hekyl commented Mar 23, 2021

HI, same here, from f to g to Ravenholm. vsanpullstatistics are not able to finish and they run may be minutes per vsan node. We'd like to test any new beta if it'll change this behavior. We'll try to reach you asu you are suggesting above, thanks

@rschitz
Copy link
Member

rschitz commented Apr 5, 2021

5001984

@rschitz rschitz closed this as completed Apr 5, 2021
rschitz added a commit that referenced this issue Jun 23, 2021
rschitz added a commit that referenced this issue Jun 27, 2021
rschitz added a commit that referenced this issue Jun 27, 2021
vmdude added a commit that referenced this issue Mar 28, 2022
* Update VIM25VsanmgmtRuntime.pm

#255

* Update VIM25VsanmgmtStub.pm

#255

* Delete VMware-vSphere-Perl-SDK-7.0.0-16815913.x86_64.tar.gz

#255

* Create VMware-vSphere-Perl-SDK-7.0.0-17698549.x86_64.tar.gz

#255

* Update VsanPullStatistics.ps1

#254

* Update ViConnect.ps1

#254

* #237

* #237

* Revert "#237"

This reverts commit 5179624.

* Revert "#237"

This reverts commit a95d642.

* Revert "Revert "#237""

This reverts commit 57122fb.

* Update exportSexiGrafBundle.sh

#254

* Update exportSexiGrafBundle.sh

* Update exportSexiGrafBundle.sh

#254

* Update importSexiGrafBundle.sh

#254

* Update export-import.php

#257

* Update VMware_All_Version.json

* Update VsanPullStatistics.ps1

* Update ViPullStatistics.pl

#259

* Update graphite.conf

* Update importSexiGrafBundle.sh

* Update ports.conf

* Update credstore.php

* Update export-import.php

* Update importSexiGrafBundle.sh

#254

* Create powershell.config.json

#261

* Update VsanPullStatistics.ps1

* Update importSexiGrafBundle.sh

* Update VsanPullStatistics.ps1

* Update VsanPullStatistics.ps1

* Update VsanPullStatistics.ps1

#249

* Update VsanPullStatistics.ps1

#174

* Update VsanPullStatistics.ps1

#174

* Update VsanPullStatistics.ps1

#174

* Update VsanPullStatistics.ps1

#143

* Update VMware_vSAN_Monitor_TCP.json

#250

* Derivative to nonNegativeDerivative

#265

* Update FreeNAS.json

* Update VMware_All_Datastore_Usage_Distribution.json

* Update VMware_All_Version.json

* Update VMware_vCenter_Bad_Events.json

* Update powershell.config.json

* Update VsanPullStatistics.ps1

* Update importSexiGrafBundle.sh

* Update addVsanCrontab.sh

* Update credstore.php

* Update ViConnect.ps1

* Update VMware_Cluster_FullStats.json

http://obfuscurity.com/2013/05/Graphite-Tip-Counting-Number-of-Metrics-Reported

* Update VMware_All_Cluster_FullStats.json

http://obfuscurity.com/2013/05/Graphite-Tip-Counting-Number-of-Metrics-Reported

* Create ViPullStatistics.ps1

* Update ViPullStatistics.ps1

* Update ViConnect.ps1

* Update ViPullStatistics.ps1

#182

* Update ViPullStatistics.ps1

#254

* Update ViPullStatistics.ps1

#143

* Update VMware_All_Version.json

* Update ViPullStatistics.ps1

* Update ViPullStatistics.ps1

#272

* Update ViPullStatistics.ps1

#272

* Update VsanPullStatistics.ps1

* Update VsanPullStatistics.ps1

#143

* Update ViPullStatistics.ps1

* Update ViPullStatistics.ps1

#143

* Update ViPullStatistics.ps1

#143

* Update ViPullStatistics.ps1

#254 #143

* Update VMware_VSAN_Resync.json

* Update VMware_All_Version.json

* Update VMware_Multi_Cluster_Capacity_Planning.json

* Update VMware_Multi_Cluster_QuickStats.json

* Update ViOfflineInventory

* Create VmNetInfos.csv

#269

* #269

* Update ViPullStatistics.ps1

#254

* Update VMware_Multi_Cluster_QuickStats.json

* Update VMware_Cluster_Multi_ESX_LiteStats.json

* Update VMware_All_Cluster_SuperStats.json

* Update VMware_All_Cluster_FullStats.json

* Update ViPullStatistics.ps1

#254

* Update ViPullStatistics.ps1

#254

* Update ViPullStatistics.ps1

#254

* Update ViPullStatistics.ps1

#254

* #192

CpuWait to CpuIoWait

* Update ViPullStatistics.ps1

* #203

numXVMotion

* #269

* Update ViVmInventory.ps1

#269

* Update ViVmInventory.ps1

#269

* Update ViVmInventory.ps1

* Update ViVmInventory.ps1

* #269

* #269

* Update ViPullStatistics.ps1

* Update VsanPullStatistics.ps1

* #269

* #254

* Update ViOfflineInventory.ps1

* Update addVsanCrontab.sh

* Update ViPullStatistics.ps1

#254

* Update ViPullStatistics.ps1

#254

* #254

* Update ViPullStatistics.ps1

* Update ViPullStatistics.ps1

* Update SexiGraf_Pull_Exec_Time.json

* Update updateInventory.sh

* Update refresh-inventory.php

* Delete PullGuestInfo.sh

* Update ViPullStatistics.ps1

* Update ViOfflineInventory.ps1

* Create PullGuestInfo.sh

* #269

* Update credstore.php

* Update VMware_All_Version.json

* Update ViOfflineInventory.ps1

* Update collectd.conf

* Update ViOfflineInventory.ps1

* Update refresh-inventory.php

* Update ViPullStatistics.ps1

* Update ViOfflineInventory.ps1

#269

* Update PullGuestInfo.sh

* Update ViOfflineInventory.ps1

* Update ViPullStatistics.ps1

* Update ViPullStatistics.ps1

* Update VsanPullStatistics.ps1

* Update ViOfflineInventory.ps1

* Update ViPullStatistics.ps1

* Update storage-schemas.conf

* Update importSexiGrafBundle.sh

* Update ViPullStatistics.ps1

fix snapshot count for vsan

* #254

* Update WhisperAutoPurge.ps1

* #254

* Update refresh-inventory.php

* Update importSexiGrafBundle.sh

#254

* Update ViOfflineInventory

* #239

* #229

* #269

* Update VsanPullStatistics.ps1

* Update ViOfflineInventory.ps1

* #276

* #276

* #277

* Update PullGuestInfo.sh

* Update VsanPullStatistics.ps1

* Update VMware_BroStats.json

* #221

* Update ViPullStatistics.ps1

* Update VsanPullStatistics.ps1

* Update VMware_BroStats.json

* Update VMware_VM_Inventory.json

* Update VMware_ESX_Inventory.json

* Update export-import.php

* Update VMware_BroStats.json

* Update VMware_VM_Inventory.json

* Update VMware_ESX_Inventory.json

* Update SexiGraf_Pull_Exec_Time.json

* Update storage-schemas.conf

* Update ViPullStatistics.ps1

* Update VMware_Cluster_FullStats.json

* Update VMware_Cluster_Multi_ESX_LiteStats.json

* Update VsanPullStatistics.ps1

* #203

* Update ViPullStatistics.ps1

#264

* Update importSexiGrafBundle.sh

* Update VsanPullStatistics.ps1

#264

* #264

* Update sexigraf

* Update VsanPullStatistics.ps1

#264

* Update VMware_All_Cluster_Top_N_VM_Stats.json

* Update VMware_vSAN_Monitor_TCP.json

* #264

* Update ViPullStatistics.ps1

#264

* Update addVsanCrontab.sh

* Update ViPullStatistics.ps1

#264

* Update VMware_vSAN_SMART_Stats.json

* Update ViPullStatistics.ps1

* Update ViPullStatistics.ps1

* Update ViPullStatistics.ps1

* Update sexigraf

* Update carbon.conf

* Update sexigraf

* Update ViPullStatistics.ps1

* Update ViPullStatistics.ps1

* Update ViPullStatistics.ps1

* Update ViPullStatistics.ps1

* Update ViPullStatistics.ps1

* Update ViPullStatistics.ps1

* Update storage-schemas.conf

* Update ViPullStatistics.ps1

#264

* Update addVsanCrontab.sh

#264

* Update ViPullStatistics.ps1

#250

* Update ViPullStatistics.ps1

* Update importSexiGrafBundle.sh

* #264

* #264

* Delete log4perl.conf

* Update LICENSE.md

* Update storage-schemas.conf

* Update importSexiGrafBundle.sh

* Update importSexiGrafBundle.sh

* Update README.md

* Update graphite.conf

grafana/grafana#45117 (comment)

* Update graphite.conf

* Create netdata.conf

* Update ViPullStatistics.ps1

* Update importSexiGrafBundle.sh

* Update VMware_All_Cluster_SuperStats.json

* Update ViPullStatistics.ps1

* Update ViPullStatistics.ps1

* Update export-import.php

* Update ViPullStatistics.ps1

* Update VMware_Multi_VSAN_Monitor.json

* Update VMware_VSAN_Monitor.json

* Update VMware_Multi_Cluster_Top_N_VM_QuickStats.json

* Update VMware_Multi_Cluster_Top_N_VM_Overcommit.json

* Update VMware_All_Standalone_ESX_VM_Stats.json

* Update VMware_All_Standalone_ESX_Top_N_VM_Stats.json

* Update VMware_All_Cluster_VM_Stats.json

* Update VMware_All_Cluster_Top_N_VM_Stats.json

* Update ViPullStatistics.ps1

* Update Netdata.json

* Update VMware_Multi_Cluster_Top_N_VM_Latency.json

* Update ViPullStatistics.ps1

#287

* Update ViPullStatistics.ps1

* Update ViPullStatistics.ps1

* Update index.php

* Update ViOfflineInventory.ps1

* Update ViPullStatistics.ps1

#268

* Update ViOfflineInventory.ps1

* #268

* Update ViPullStatistics.ps1

#159

* Update VMware_VSAN_Resync.json

* Update VMware_Multi_VSAN_Monitor.json

* #250

* Update README.md

* Update ViOfflineInventory.ps1

* Update VMware_BroStats.json

* Update ViOfflineInventory.ps1

* Update SexiGraf_Web_Admin.json

* Update PullGuestInfo.sh

#211

* Update VMware_VSAN_Resync.json

* Update ViPullStatistics.ps1

* Update ViPullStatistics.ps1

* Update ViOfflineInventory.ps1

* Update VMware_All_Cluster_SuperStats.json

* Update VMware_Cluster_FullStats.json

* Update VMware_All_Cluster_SuperStats.json

* Update VMware_All_Cluster_SuperStats.json

* Update VMware_Multi_vSAN_Monitor_TCP.json

Co-authored-by: Raphaël SCHITZ <raphael@schitz.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants