Smart view of unlimited storage

[HDembinski]

Currently, we convert the unlimited storage to double whenever a view is requested. Can the smart views introduced for accumulators used to avoid this? The view would have to keep a pointer to the unlimited storage type id, and always return the dtype which matches the type id.

Conversion to double would then be only necessary when the unlimited storage is in the unlimited mode, which basically never happens.

[henryiii]

We can't do this as an ndarray, because the memory can't move around and dtypes are static (I'm 85% sure). We could build a smart .view() adaptor that could track this, though you could still not do np.asarray(h.view()) and have that not break if the histogram was filled.

[henryiii]

PS: If not clear from above, I'm not against a smart view, just pointing out the potential issues. As long as it's clear that the view is temporary on fills, it could be useful. (And if we can come up with a way to invalidate the (numpy) memoryview or change it, that would be great)

[HDembinski]

We can't do this as an ndarray, because the memory can't move around and dtypes are static (I'm 85% sure).

What about subclassing ndarray?

np.asarray(h.view())

There is np.asanyarray. It is a reasonable price to pay. I was happy with the original solution of converting to double, because it can't be broken at all, but if you care about unlimited actually handing out integers, then we need a better solution.

[HDembinski]

A smart view could delay the conversion to double until its __array_interface__ is called or whatever np.asarray queries. Then you could have methods to get the original integers from the view and you get everything at the cost of having to call a method to access the integers.

[HDembinski]

Yes, asarray calls the __array_interface__ so that should work.

[HDembinski]

Any comments on the ansatz:

View is implemented as a subclass of ndarray, with a dynamic dtype and a pointer to the original unlimited storage. Calling ufuncs on this ndarray should then work normally, also assignment.

__array_interface__ is overloaded on this ndarray as a dynamic property. When called, it creates a new array with dtype double as the buffer, fills it with the content of unlimited storage. So np.asarray makes a copy and leaves the original unlimited storage in its original state.

Conversion to double of unlimited now never happens. When the buffer of the unlimited storage is in the unlimited stage, we use the dtype np.void. This means that the view probably cannot be assigned to, but you can still use np.asarray on it. We should be able to make it so that the operators and ufuncs still work. Remember that this stage is very hard to reach in practice.

[henryiii]

I had a couple of ideas - this was, as far as I understand it, roughly one of them. You probably could select the dtype to provide on access, I would expect so, anyway. The caveat here is large, as it would would both break the no-copy promise of view, and would break histogram value and array setting (since it uses a view). We could work around the latter with a temporary view that was not public, though. Maybe it could be an option with .view(), with the current one being default? I need to look into the options we get with memoryviews and ndarray subclasses, and get back to this after 0.6.0.

My second idea would be much harder on the boost-histogram level, but might work. If we could keep a std::shared_pointer to each block of memory that boost-histogram allocates, then we could give that shared pointer to PyBind11, and it would keep the old memory from being removed when unlimited makes a jump if a Python pointer to it still existed. Then, the only caveat would be that a view is not guaranteed to be valid after a fill (or other modification), but wouldn't segfault Python if that happened. You could see and modify the values from Python, check the dtype, etc. As for performance, boost-histogram wouldn't need to use the shared interface (since it already knows when memory has been allocated), avoiding a performance penalty on fills. Would something like be possible, probably using a subclass of unlimited?

[HDembinski]

I don't like the shared_ptr idea. My ansatz is not breaking the no-copy promise of view, maybe you should read it again more carefully? You get a copy when you call np.asarray on the view. The new view itself is never triggering a change in unlimited.

[HDembinski]

If you hold a view to some complex object, whose interface structure may change, you should not expect it to be "constant". I think this is clear for people. I think my ansatz always shows a valid numpy array when you query the view, although it may be a different array when you query at different times.

[henryiii]

It does look better/safer than when I first looked at it. At first I felt it was just delaying the issue. It still makes a copy any time someone uses it (other than single element access), but most of the time when you are using Numpy, you are making copies, so that's not that bad. A few things probably won't work quite like a numpy array, like in place ufuncts, but I don't think that's a deal-breaker. And since we control the object, we possibly can add support for some of these situations if users do run into problems (I can't currently imagine a use for an in-place ufunc on one, for example). And if you loop over massive numbers (or even small numbers) of histograms, you can perform computations, then go on, throwing away any extra copies, and never explode to doubles.

I'm in.

[henryiii]

(Being different at different times is fine, just not segfaulting ;) )

[HDembinski]

Sure, no segfaulting allowed. As far as I can see, ufunc's should work except when the view is in the unlimited mode, but that basically never happens. In other cases, there wouldn't be a need to make a copy when a ufunc is using the view. The view will not change while the ufunc is running, unless you are filling the histogram at the same time in a parallel thread. And like you said, numpy does not protect you from that scenario either.