cum_to_incr() produces cumulative values when NaNs exist above the diagonal

[genedan]

Are you on the latest chainladder version?

• Yes, this bug occurs on the latest version.

Describe the bug in words

Found during investigation of #692. When there are missing values in the upper left-hand corner of the triangle, as is often the case when data from older years are not available, cum_to_incr() retains cumulative values in valuation periods immediately following the NaNs.

How can the bug be reproduced?

df = pd.DataFrame(data={
    'origin': [2022, 2022, 2023],
    'development': [2022, 2023, 2023],
    'reported': [np.nan, 222000, 78000]}
)

tri_from_df = cl.Triangle(
    data=df,
    origin='origin',
    development='development',
    columns=['reported'],
    cumulative=True
)

print(tri_from_df)

which prints:

           12        24
2022      NaN  222000.0
2023  78000.0       NaN

Now execute:

tri_from_df.cum_to_incr()

Which prints:

           12        24
2022      NaN  222000.0
2023  78000.0  -78000.0

The mysterious -78000.0 value was documented in #692. What's new here is that the 2022 origin year evaluated as of 24 months is 222000, which is the cumulative value at that time. This should actually be a NaN, since we do not know the value as of the previous valuation of 12 months, i.e., 222000.0 - NaN should be NaN.

This issue also happens in the more well-known xyz sample:

xyz = cl.load_sample('xyz')
xyz
Out[9]: 
                                       Triangle Summary
Valuation:                                      2008-12
Grain:                                             OYDY
Shape:                                   (1, 5, 11, 11)
Index:                                          [Total]
Columns:    [Incurred, Paid, Reported, Closed, Premium]
xyz['Incurred']
Out[10]: 
          12       24       36       48       60       72       84       96       108      120      132
1998      NaN      NaN  11171.0  12380.0  13216.0  14067.0  14688.0  16366.0  16163.0  15835.0  15822.0
1999      NaN  13255.0  16405.0  19639.0  22473.0  23764.0  25094.0  24795.0  25071.0  25107.0      NaN
2000  15676.0  18749.0  21900.0  27144.0  29488.0  34458.0  36949.0  37505.0  37246.0      NaN      NaN
2001  11827.0  16004.0  21022.0  26578.0  34205.0  37136.0  38541.0  38798.0      NaN      NaN      NaN
2002  12811.0  20370.0  26656.0  37667.0  44414.0  48701.0  48169.0      NaN      NaN      NaN      NaN
2003   9651.0  16995.0  30354.0  40594.0  44231.0  44373.0      NaN      NaN      NaN      NaN      NaN
2004  16995.0  40180.0  58866.0  71707.0  70288.0      NaN      NaN      NaN      NaN      NaN      NaN
2005  28674.0  47432.0  70340.0  70655.0      NaN      NaN      NaN      NaN      NaN      NaN      NaN
2006  27066.0  46783.0  48804.0      NaN      NaN      NaN      NaN      NaN      NaN      NaN      NaN
2007  19477.0  31732.0      NaN      NaN      NaN      NaN      NaN      NaN      NaN      NaN      NaN
2008  18632.0      NaN      NaN      NaN      NaN      NaN      NaN      NaN      NaN      NaN      NaN
xyz['Incurred'].cum_to_incr()
Out[11]: 
          12       24       36       48      60      72      84      96     108    120   132
1998      NaN      NaN  11171.0   1209.0   836.0   851.0   621.0  1678.0 -203.0 -328.0 -13.0
1999      NaN  13255.0   3150.0   3234.0  2834.0  1291.0  1330.0  -299.0  276.0   36.0   NaN
2000  15676.0   3073.0   3151.0   5244.0  2344.0  4970.0  2491.0   556.0 -259.0    NaN   NaN
2001  11827.0   4177.0   5018.0   5556.0  7627.0  2931.0  1405.0   257.0    NaN    NaN   NaN
2002  12811.0   7559.0   6286.0  11011.0  6747.0  4287.0  -532.0     NaN    NaN    NaN   NaN
2003   9651.0   7344.0  13359.0  10240.0  3637.0   142.0     NaN     NaN    NaN    NaN   NaN
2004  16995.0  23185.0  18686.0  12841.0 -1419.0     NaN     NaN     NaN    NaN    NaN   NaN
2005  28674.0  18758.0  22908.0    315.0     NaN     NaN     NaN     NaN    NaN    NaN   NaN
2006  27066.0  19717.0   2021.0      NaN     NaN     NaN     NaN     NaN    NaN    NaN   NaN
2007  19477.0  12255.0      NaN      NaN     NaN     NaN     NaN     NaN    NaN    NaN   NaN
2008  18632.0      NaN      NaN      NaN     NaN     NaN     NaN     NaN    NaN    NaN   NaN

Where the values of 11171.0 at the 1998 AY, 36 valuation and the 13255.0 at the 1999 AY 24 valuation are the original cumulative, not incremental values.

What is the expected behavior?

tri_from_df.cum_to_incr()

should result in:

           12  24
2022      NaN NaN
2023  78000.0 NaN

[danielfong-act]

Triangle-level arithmetic* treats NaNs exactly like zeros and I think this issue is more of an unintended consequence of that design decision than a bug. This got a bit of discussion about this in #639, and I also think this affects #694 and #692 (in part). In short, NaNs have to be indistinguishable from 0 in order to support sparse triangles.

This leads to an issue here, however, because there's no way to know if (2022,12) is a missing value or a cumulative zero, so there's no way to logically know how to make the conversion-- unless we added some missing value flag or whatever.

* any arithmetic involving a triangle and some object that can be broadcast

edit: Triangle level arithmetic only sometimes treats zeros like NaNs. It seems like multiplication and division treat NaN like NaN, while addition and subtraction treat it like a zero. Since we're dealing with the latter two operations here, I think that the point still stands.