no data

[n0405]

Hi ,

there is one problem, which I noticed late. There is no data from 22 June. I don't have idea what is happens. this applies to all vCSA .

[rschitz]

hi, looks like graphite is dead. can you check disk space on your sexigraf appliance ?

[n0405]

~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 797M 836K 796M 1% /run
/dev/mapper/vgsexigraf-root 15G 7.1G 6.5G 53% /
tmpfs 3.9G 180K 3.9G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda1 511M 4.0K 511M 1% /boot/efi
/dev/sdb1 30G 21G 7.5G 74% /mnt/wfs
tmpfs 797M 0 797M 0% /run/user/0

[rschitz]

ok, did you try to restart the appliance ?

[n0405]

yes , twice ,
he is log (carbon-relay-a/clients.log)
26/06/2023 13:11:37 :: connecting to carbon daemon at 127.0.0.1:2104:a
26/06/2023 13:11:37 :: connecting to carbon daemon at 127.0.0.1:2204:b
26/06/2023 13:11:37 :: CarbonClientFactory(127.0.0.1:2104:a)::startedConnecting (127.0.0.1:2104)
26/06/2023 13:11:37 :: CarbonClientFactory(127.0.0.1:2204:b)::startedConnecting (127.0.0.1:2204)
26/06/2023 13:11:37 :: CarbonClientProtocol(127.0.0.1:2104:a)::connectionMade
26/06/2023 13:11:37 :: CarbonClientFactory(127.0.0.1:2104:a)::connectionMade (CarbonClientProtocol(127.0.0.1:2104:a))
26/06/2023 13:11:37 :: Destination is up: 127.0.0.1:2104:a
26/06/2023 13:11:37 :: CarbonClientFactory(127.0.0.1:2204:b)::clientConnectionFailed (127.0.0.1:2204) Connection was refused by other side: 111: Connection refused.
26/06/2023 13:11:37 :: Destination is down: 127.0.0.1:2204:b (1/5)
26/06/2023 13:11:39 :: CarbonClientFactory(127.0.0.1:2204:b)::startedConnecting (127.0.0.1:2204)
26/06/2023 13:11:39 :: CarbonClientProtocol(127.0.0.1:2204:b)::connectionMade
26/06/2023 13:11:39 :: CarbonClientFactory(127.0.0.1:2204:b)::connectionMade (CarbonClientProtocol(127.0.0.1:2204:b))
26/06/2023 13:11:39 :: Destination is up: 127.0.0.1:2204:b

[rschitz]

what about /var/log/sexigraf/graphite/.log and /var/log/apache2/graphite/.log ?

[n0405]

in /var/log/apache2/graphite/ there are two files /error.log which is empty, and access which don't have any error.

in var/log/sexigraf/graphite/ . also two logs, exeption.log has time stamp form May.

and info.log:
2023-06-26,13:53:00.513 :: Invalid parameters (Missing required parameter 'query')
2023-06-26,13:53:01.345 :: Invalid parameters (Missing required parameter 'query')
2023-06-26,13:53:01.496 :: Invalid parameters (Missing required parameter 'query')
2023-06-26,13:53:02.026 :: Invalid parameters (Missing required parameter 'query')
2023-06-26,13:54:49.848 :: find_view query=vmw.* local_only=0 matches=5
2023-06-26,13:54:49.848 :: received remote find request: pattern=vmw.* from=1687784088 until=1687787690 local_only=0 format=treejson matches=5
2023-06-26,13:54:49.934 :: find_view query=vmw.. local_only=0 matches=7
2023-06-26,13:54:49.934 :: received remote find request: pattern=vmw.. from=1687784088 until=1687787690 local_only=0 format=treejson matches=7
2023-06-26,13:54:49.972 :: find_view query=vmw...* local_only=0 matches=7
2023-06-26,13:54:49.973 :: received remote find request: pattern=vmw...* from=1687784088 until=1687787690 local_only=0 format=treejson matches=7
2023-06-26,13:54:50.014 :: find_view query=vmw...AAAAAA.esx.* local_only=0 matches=10
2023-06-26,13:54:50.014 :: received remote find request: pattern=vmw...AAAAAA.esx.* from=1687784089 until=1687787691 local_only=0 format=treejso
n matches=10

2023-06-26,13:54:55.971 :: find_view query=vmw.* local_only=0 matches=5
2023-06-26,13:54:55.971 :: received remote find request: pattern=vmw.* from=1687784095 until=1687787697 local_only=0 format=treejson matches=5
2023-06-26,13:54:55.991 :: find_view query=vmw.. local_only=0 matches=7
2023-06-26,13:54:55.991 :: received remote find request: pattern=vmw.. from=1687784095 until=1687787697 local_only=0 format=treejson matches=7
2023-06-26,13:54:56.052 :: find_view query=vmw...* local_only=0 matches=7
2023-06-26,13:54:56.052 :: received remote find request: pattern=vmw...* from=1687784095 until=1687787697 local_only=0 format=treejson matches=7
2023-06-26,13:54:56.071 :: find_view query=vmw...cisco-cloud.esx.* local_only=0 matches=10
2023-06-26,13:54:56.071 :: received remote find request: pattern=vmw...cisco-cloud.esx.* from=1687784095 until=1687787697 local_only=0 format=treejson matches=10
2023-06-26,13:55:04.005 :: find_view query=vmw.* local_only=0 matches=5
2023-06-26,13:55:04.006 :: received remote find request: pattern=vmw.* from=1687784103 until=1687787705 local_only=0 format=treejson matches=5
2023-06-26,13:55:04.026 :: find_view query=vmw.. local_only=0 matches=7

[rschitz]

in /var/log/sexigraf/carbon/carbon-/.log ?

[n0405]

in console.log I find this :
27/06/2023 00:00:54 :: Invalid schemas found in sexigraf: A Whisper database may not be configured having two archives with the same precision (archive1: (300, 576), archive2: (300, 2016))
27/06/2023 00:00:54 :: Invalid schemas found in smart: A Whisper database may not be configured having two archives with the same precision (archive1: (300, 576), archive2: (300, 2016))
27/06/2023 00:00:54 :: Invalid schemas found in anything_else: A Whisper database may not be configured having two archives with the same precision (archive1: (300, 576), archive2: (300, 2016))

[rschitz]

can you show me the content of /opt/graphite/conf/storage-schemas.conf ? did you changed something in this file?

[n0405]

yes I did ,
nohup /usr/bin/find -L /mnt/wfs/whisper/vmw/ -type f ( -name '*.wsp' ) -exec /usr/local/bin/whisper-resize.py {} 1m:24h,1m:48h,5m:7d,60m:30d,60m:90d,60m:1y --nobackup ; &

[carbon]
pattern = ^carbon.
retentions = 60:90d

[sexigraf]
pattern = ^collectd.sexigraf.
#retentions = 1m:1h,2m:2h,4m:3h,12m:12h,24m:2d,96m:7d,192m:30d,3840m:1y
retentions = 1m:24h,5m:48h,5m:7d,5m:30d,5m:90d,5m:1y,60m:2y,120m:5y
[smart]
pattern = ^smart.
#retentions = 60m:7d,240m:30d,720m:90d,2880m:1y,5760m:2y,17280m:5y
retentions = 1m:24h,5m:48h,5m:7d,5m:30d,5m:90d,5m:1y,60m:2y,120m:5y

[pfsense]
pattern = ^pfsense.
retentions = 1m:1h,2m:2h,4m:3h,12m:12h,24m:2d,96m:7d,192m:30d,3840m:1y

[vsan_health]
pattern = .*.vsan.ObjectHealthDetail.
retentions = 5m:24h,10m:48h,60m:7d,240m:30d,720m:90d,2880m:1y

[vsan_sync]
pattern = .*.vsan.SyncingVsanObjects.
retentions = 5m:24h,10m:48h,60m:7d,240m:30d,720m:90d

[vsan_smart]
pattern = .*.vsan.disk.smart.
retentions = 5m:24h,10m:48h,60m:7d,240m:30d,720m:90d,2880m:1y

[vsan_spaceDetail]
pattern = .*.vsan.spaceDetail.
retentions = 5m:24h,10m:48h,60m:7d,240m:30d,720m:90d,2880m:1y,5760m:2y,17280m:5y

[vsan]
pattern = ^vsan.
retentions = 5m:12h,10m:24h,20m:48h,80m:7d,320m:30d,640m:90d

[anything_else]
pattern = .*
retentions = 1m:24h,5m:48h,5m:7d,5m:30d,5m:90d,5m:1y,60m:2y,120m:5y

#retentions = 5m:24h,10m:48h,60m:7d,240m:30d,720m:90d,2880m:1y,5760m:2y,17280m:5y

[rschitz]

you can't do that: 1m:24h,5m:48h,5m:7d,5m:30d,5m:90d,5m:1y,60m:2y,120m:5y
you have to do: 1m:24h,5m:1y,60m:2y,120m:5y
also, 1m:24h,1m:48h,5m:7d,60m:30d,60m:90d,60m:1y would be 1m:48h,5m:7d,60m:1y
but keep in mind that it might cost you a lot of storage