Network Graph (sometimes wrong algebraic sign for download/upload with high peaks)

[deMattin]

Sometimes in the RR-Database the value seems to be saved with the wrong algebraic sign.
For me, it was the 2nd time in 2 weeks now.
It then looks like this:

I've made a zoomed 14 days view to see it better, because the peak is much higher in dayview. In dayview I'm able to see, that there are two values with this wrong sign.
The peak is as high as 250 MByte/sec in dayview with the 10 second normalized values! That's much more, than the ethernet interface is able to transfer.

I have an idea, how to fix the wrong sign, but before I want to know, if anybody else has seen this, too.

The fix would be easy, I think:
It should be possible to force the alebraic sign like this:

1. Read the value
2. Eliminate any sign with "abs"-function.
3. force "-" to onload and "+" for upload.
4. save value in rrd

If the error is in the RRD readout, this can be done in the readout routine (also).
But I don't think, that the problem is in the readout, because in this case a newly generted graph (reload) would have fixed this.
An implementation in the readout routine would fix the view but not the wrong values in the database.

Would not be hard to implement this and I would develop a patch for this - but only, if I'm not alone with this problem.

And does anyone have an idea, where this wrong readout values may be come from (sign and much to high value)?

greets,
Martin

Edit:

I had a look at the rrd definitions and tools and the problem with peaks is a problem with many rrd installations.
A often given tip is to define a max and min value for the values and if a value is out of range, it then will be dropped.
I have searched in the repo but haven't yet found where it is possible to define this.

I think, this would be the best way to prevent the problems with wrong signs (without any reproduceable reason) and possible range overflows if the 2nd value is smaller than the previous one (tx/rx value is only possible max of 2^32 = 4GByte) which is probably causing the sign issue!
Today I made a 6 GB upload to my ownCloud and there was a wrong peak again ...

[ajs124]

I haven't looked at my network graphs in a while, but now that bring it up, I did and this seems to affect me as well.

[deMattin]

RRD creation is defined hardcoded in "rpimonitord" as "DS:$name:$type:600:U:U".
With the parameter ":U:U" meaning, that the min and max values are set to unlimited, which is universal but causes the problems, if you get a range overflow in source values (values starting with 0 again - same problem, if you reboot the RPi).

I think, we have to make this parameter (all parameters?) changeable in the "rpimontor.conf" with a new set of defnitions.
When you want to save B/s values, it would be better to save the values always with a positive sign as read out of the source and define it as ":0:12500000" instead (the rPi 100MBit/s interface has max 12,500,000 bytes per second, which is 100,000,000 bits per second).
In this case a range overflow will be ignored (undefined value) or "guessed" and corrected (investigation needed).

Another more simple approach would be to define the rrd as "counter" instead of "derive" in rpimonitor.conf for the Bytes/s value.
In this case only increasing of values is allowed and the sign has to be set later in the graph definition.
But I have read, that this may cause problems with B/s and have to investigate here further more.
http://apfelboymchen.net/gnu/rrd/create/
and a tutorial at:
http://www.vandenbogaerdt.nl/rrdtool/tutorial/rrdcreate.php

Best would be a combination of both, I think!
The best way to create the rrd has to be investigated and after this, it should be no big problem to patch this.

Today I have no time to investigate it any more but I'll try to get it the next days ...

greets,
Martin

PS: Same problem is with the static value for the net up/down in static.
Every 4 GB the value starts with 0 again.
I will have a look at a better source for net-static-values - maybe with the help of vnstat (I have installed it anyway).

Edit: Replace "gauge" with "strike" in third paragraph.

[XavierBerger]

When I first read the description of the issue I first think about a counter reaching its maximum and restarting from 0. I already saw that in some Cisco monitoring.
Fixing this in RRD should be possible by setting the min/max values as proposed upper.
But, what about the counter displayed in the status page? I feel you should see it restarting from 0. Managing this will certainly be more complex since it will require the management of some kind of "steps"...
The idea to find another counter is a good workaround. I already use it for Cisco by using a 64 bits counter to push the issue out... This would be my preferred solution for short term. The idea to allow min/max configuration would be a good next step.

[deMattin]

At first approach, today I defined the rrd as "counter" instead of "derive" in rpimonitor.conf for the rx/tx Bytes/s values.
Seems to work without any further changes.
I will have a look, how it will behave the next time there's a range overflow.

[deMattin]

Made some tests and only changing to "derive" doesn't work because the rrd database saves the type of data saving.

So I suggest to use COUNTER type with positive values for up and down and change it in default.conf to:

dynamic.10.name=net_received
dynamic.10.source=/sys/class/net/eth0/statistics/rx_bytes
dynamic.10.regexp=(.*)
dynamic.10.postprocess=
dynamic.10.rrd=COUNTER

dynamic.11.name=net_send
dynamic.11.source=/sys/class/net/eth0/statistics/tx_bytes
dynamic.11.regexp=(.*)
dynamic.11.postprocess=
dynamic.11.rrd=COUNTER

Then stop the rpimonitor (sudo service rpimonitor stop)

Then delete the rrd-files for send and receive (net_received.rrd and net_send.rrd in /usr/share/rpimonitor/web/stat/), so that the RRDs are created new as COUNTER type.

Then start the rpimonitor again (sudo service rpimonitor start)

After all you get this (after some down- and uploads):

I have not yet found a way to negate the rrd output to get the negative download graph as before but I don't think it is so bad this way with up and down both in same direction in the graph (maybe some people like it even more - as I do - because of the better scaling).
And in this recorded session there were two 4GB-Range overflows and you see, there are NO PEAKS any more.
It seems not to be necessary to set a min or max value!
It works as expected and max/min value would only be an additional fix for wrong readouts and perhaps wrong reboot values.
But it should be possible to set max and min in future and I will have to try this later on.

So for now I would suggest to change it as above mentioned.
The advantage of this solution is, that there is no further patch necessary and there is no further system load from additional programs.

For up and down on the status screen there seems to be the best way to use the help of further software like vnstat if one wants to fix this.
Vnstat is light weight and gives everything we need here (even more, like traffic per month and day)
But this is a further dependency and so it should be optional, I think.
The next days I will make a suggestion, how this may be implemented.

greets,
Martin

[XavierBerger]

min and max values are now implemented in RRD and fix the issue. Closing.