Increase storage retention storage-schema.conf

[bbaiq]

Hi,
we 'd like to increase storage retention values, in order to be able to troubleshoot/compare from past week (or past 2 weeks) to fresh sampling period.

something like that
from /etc/carbon/storage-schema.conf
#retentions = 1m:1h,2m:2h,4m:3h,12m:12h,24m:2d,96m:7d,192m:30d,3840m:1y
to
#retentions = 1m:1d,2m:2h,4m:3h,8m:1d,16m:7d,32m:14d,64m:30d,192m:60d,3840:1y

Is there any other policy we have to take care of in order to maintain performance ?

One carbon-cache.py instance may not be enough to handle the I/O load ? How to scale out the number of carbon-cache instance ?
Thanks a lot

We love Sexigraf since 4 years !! Thanks a lot !!

[rschitz]

Hi, thanks a lot for your amazing support!
First, only the vSAN metrics are pulled every minutes, the regular ones (vcenter/esx) are pulled every 5 minutes so you can't go under that for your carbon conf

Second, you can change the carbon conf but it will only affect new created files (i.e. new vms) check this command if you want to apply a new schema on existing files #217 (comment)

In the last sexigraf version, you got 2 carbon-cache instances which proved to be more than enough for 200000+ VMs, 7000+ ESX and 60+ vCenter. If you want to increase the performances at this point i suggest you to look at storage option like using ramdrive and rsync every night on non volatile storage to avoid losing everything.

The main concern here will be the storage space unless you have a very small inventory. try to apply the policy to one file to get an idea of the size and multiply this by 10 and by your vms total to guess the space you'll need.

[bbaiq]

Hi, Thanks a lot ! Perfect answer. I ran the cmd online (scoping with only one cluster) in order to apply new schema on existing wsf files. (maybe I should run it with stop carbon-cache service before) ? nohup /usr/bin/find -L /mnt/wfs/whisper/vmw/vcenter01_hbgt_intra/adista/my-clr1/ -type f \( -name '*.wsp' \) -exec /usr/local/bin/whisper-resize.py {} 5m:24h 10m:14d 80m:30d 240m:90d 720m:120d 2880m:1y 5760m:2y 17280m:5y --nobackup \; & So I will see the storage needs. Do I need also to add pattern for the new files in storage-schemas.conf ? [Myclr1] pattern = ^vmw/vcenter01_hbgt_intra/adista-ncy/my-clr1/.* retentions = 5m:24h 10m:14d 80m:30d 240m:90d 720m:120d 2880m:1y 5760m:2y 17280m:5y and when the service restarted, 'too many values to unpack (expected 2) in section [Myclr1] in /opt/graphite/conf/storage-schemas.conf' Could you please give me some advices ? Thanks a lot Benjamin

[rschitz]

before running the command be aware that you'll got a lot of "holes" in your graf until you got enough data because you change the shema and asked for more data points that you dont have yet and i'm not sure it is mandatory to stop the carbon service but i'd do it if i were you

you'll need a new patern of course until you change the main one and the retentions values are separated by comas "," not spaces ;)

[bbaiq]

by comas...my bad ;-) !! Have a good day !

[rschitz]

you're welcome ;)
let us know your experience in this process since the retention looks pretty high

[bbaiq]

the cmd for the existing points was run on the fly. some holes for the last 30d (bw 01/20 and 02/01) but it looks good for the last 14d ;-)

The storage needs are x3 compared to the last storage schemas.

[rschitz]

looks good indeed, x3 sounds ok to me. thanks for the feedback!