Regression with Mixed Effects using OnlineStats.jl?

[skanskan]

How can I run a regression with random effects using OnlineStats.jl?

For example I can use MixedModels.jl for this toy example:

N=3000
x1 = repeat(1:N, outer=N)
x2 = repeat(1:N, inner=N)
x3 = sqrt.(repeat(1:N^2))
x1x2 = x1 .* x2
gg = repeat(1:5,inner=trunc(Int,N^2/5))
y = 1 .- 2x1 + 3x2 + 0.5*x1x2 + rand(N^2) + x3.*rand(N^2)
data = DataFrame(x1=x1, x2=x2, x3=x3, x1x2=x1x2, y=y,gg=gg)
categorical!(data, :gg)
fit(LinearMixedModel, @formula(y ~x1+x2+x3+x1x2 + (1|gg)), data)

But if I increase N to 30000 or my model is more complex they it gives an error.
How can I run it streaming the data to the disk or with any other trick?

[joshday]

Linear mixed models can be fit via an EM algorithm, for which an online version exists: https://arxiv.org/abs/0712.4273

I won't have time to implement it, but would happily accept a PR!

[skanskan]

There is also a "Feasible algorithm for linear mixed model for massive data":
https://www.tandfonline.com/doi/abs/10.1080/03610918.2017.1307395

[skanskan]

Then we need the algorithm to be implemented on OnlineStats? Can't we just force any function/package to use OnlineStats to perform all its operations?

[joshday]

I'm not sure what you're suggesting, but to work with OnlineStats, something needs to implement the interface in OnlineStatsBase.