[Chain] flatten fails with different random variables in samples

[emilemathieu]

At the end of each sample function call, the samples are wrapped with a Chain object. Yet, during the construction of such an object, it is explicitly assumed that the set of random variables is fixed for all samples. Unfortunately this is not true for BNP models.

@yebai would you know how this could be extended for these cases ? What are the constraints of the Chains object's constructor ?

[yebai]

Kai and I discussed how to support this in the past. We feel the best way of supporting this is to trim the set of random variables before storing them in Chain. The interface that implements this trimming is through the return statement (see #104 (comment)). Unfortunately, this interface is not complete yet, see e.g. #81.

@xukai92 Do you remember the status of the return statement?

[yebai]

@yebai would you know how this could be extended for these cases ? What are the constraints of the Chains object's constructor ?

It’s possible to extend the Chain data structure, but I think it involves a lot of work. Since all chain related functions implicitly assume that each sample’s dimension is the same. Perhaps for now we can start by using the largest dimension of all samples, and treat missing dimensions as missing data - there are some Stats packages for handling missing data, it might be possible to combine them with Mamba so that Chain’s utility functions can correctly handle missing dimensions.

[emilemathieu]

@yebai It seems that Chain is fed with a matrix, so we could take the union of all random variable names, and fill with NaN values in samples that do not have some random variables.

[yebai]

Perhaps the following package can solve our problem here. My suggestion is that we first try to replace the matrix type of Chain with DataArray, and see whether existing Mamba utility functions can correctly work.

https://github.com/JuliaStats/DataArrays.jl

[xukai92]

DataArrays.jl (along with Missing.jl) seem to be unable to solve the problem as

import DataArrays: @data
a = Missing()
v = @data([1,2,a,4])
mean(v) # => missing

which will make Mamba.jl unable to use Chain

[yebai]

According to Missing.jl's documentation, its default behavior for missing values is as follows

"
The package defines standard operators and functions which propagate missing values: for example 1 + missing and cos(missing) both return missing.
“

We might need to adopt this package for our purpose - but this is not urgent.

[xukai92]

I think we may need another field in VarInfo called mark to mark variables as missing. This filed can also be used to solve #389 as I commented.

[yebai]

I think we may need another field in VarInfo called mark to mark variables as missing. This filed can also be used to solve #389 as I commented.

I might be wrong - but I think what @emilemathieu is proposing in #395 no longer need a marker for deleted entries from VarInfo.