BUG: pandas.core.window.rolling.Rolling.std gives all-zero output for small numbers

[JoschD]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import numpy as np
import pandas as pd

np.random.seed(123)
pd.Series(np.random.rand(10)*1e-7).rolling(3).std()

Problem description

It seems std() on a rolling window in pandas 1.2.2 rounds the numbers to exact 0.0 if the result would be < 1.0e-7.
In comparison, version 1.1.2 gave the correct result.

Expected Output

Output in Pandas 1.1.2

Out: 
0             NaN
1             NaN
2    3.417240e-08
3    2.321842e-08
4    2.407001e-08
5    6.857074e-09
6    1.861914e-08
7    1.343953e-08
8    2.564385e-08
9    1.901245e-08
dtype: float64

INSTALLED VERSIONS

commit : 2a7d332
python : 3.7.8.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.72-microsoft-standard-WSL2
Version : #1 SMP Wed Oct 28 23:40:43 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.1.2
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.3
setuptools : 47.1.0
Cython : None
pytest : 5.4.3
hypothesis : 5.20.3
sphinx : 3.1.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.16.1
pandas_datareader: None
bs4 : 4.9.3
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.0
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

Actual Output

Output in Pandas 1.2.2

Out: 
0    NaN
1    NaN
2    0.0
3    0.0
4    0.0
5    0.0
6    0.0
7    0.0
8    0.0
9    0.0
dtype: float64

INSTALLED VERSIONS

commit : 7d32926
python : 3.7.8.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.72-microsoft-standard-WSL2
Version : #1 SMP Wed Oct 28 23:40:43 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.2.2
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.3
setuptools : 47.1.0
Cython : None
pytest : 5.4.3
hypothesis : 5.20.3
sphinx : 3.1.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.16.1
pandas_datareader: None
bs4 : 4.9.3
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.0
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.5.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[mroeschke]

Thanks for the report.

This is a known limitation in the rolling aggregations and we're currently discussing on how to to best communicate it in #37051, so going to close in favor of that issue

[JoschD]

Thanks for the report.

This is a known limitation in the rolling aggregations and we're currently discussing on how to to best communicate it in #37051, so going to close in favor of that issue

Hey, sorry if I am wrong, but as far as I understand #37051 you are only rounding if differences in the method of calculating the std() to avoid dragging along numerical artifacts.
In the bug I mention there is much bigger numbers involved. All of the resulting values should be of the order of 1e-9 or 1e-8.
And the relative difference to the values of the series is even much lager. I agree, that it is probably related, but rounding at these numbers makes the whole rolling window unusable for numbers < 1.0e-7.

[mroeschke]

The code mentioned #37051 is a limit set for variances ops (which std uses), so since the floor for var is 1e-15, the effective floor for std is sqrt(1e-15) ~ 1e-8

[JoschD]

Ah, I missed the variance part. Thank you very much for the explanation!

[xmatthias]

I disagree with closing this, we are encountering the same problem while calculating mean (and std) on top of Crypto asset prices (which can become numbers down to 1e-8).

while #37051 does discuss this - it's not mentioned as a side-effect as part of the release docs for pandas 1.2 - so i would see this as an unintended regression.

A very simple example:

import pandas as pd

print(pd.__version__)

df = pd.DataFrame(data = {'data':
    [
        0.00000054,
        0.00000053,
        0.00000054,
     ]}
    )

df['mean'] = df['data'].rolling(2).mean()
df['std'] = df['data'].rolling(2).std()
print(df)

with pandas < 1.2.0, the return is as follows:

1.1.5
           data          mean           std
0  5.400000e-07           NaN           NaN
1  5.300000e-07  5.350000e-07  7.071068e-09
2  5.400000e-07  5.350000e-07  7.071068e-09

while 1.2.0 returns:

1.2.0
           data          mean  std
0  5.400000e-07           NaN  NaN
1  5.300000e-07  5.350000e-07  0.0
2  5.400000e-07  5.350000e-07  0.0

The values are nowhere near the mentioned threshold of 1e-15.