Gibbs sampling - Use blocks to reduce computation

[donhausk]

I want to use Turing for the GP-SSM model, which is based on the PGAS algorithm. The implementation is actually straight forward thanks to Turing. However, I have noticed that the Gibbs sampler runs the whole model for every subsampler. For large models this implies that there is a huge computational overhead. One way to solve this issue would be by implementing some "hacky" macros, like:

@condition_on (:x, ;y) = begin
# Do some code here which is only executed when :x or :y appears in the sampling space
end

which would be changed to something like :

if :x in spl.alg.space || :y in spl.alg.space
# Run the code above
end

This makes the model rather difficult to design because it might not be obvious what belongs inside the brackets and what not. @torfjelde had the idea to do something like the following instead :

var = @condition_on (:x, ;y) = begin
# Compute var
end

which would be changed to something like :

if :x in spl.alg.space || :y in spl.alg.space
# Compute var
cache["var"] = var
else
var = cache["var"]
end

I am anyway planning to implement this macro for my own use and I was thinking that this might be interesting for Turing in general. What do you think?

[devmotion]

It seems this is a duplicate of https://github.com/TuringLang/Turing.jl/issues/833? As @mohamed82008 mentioned there, I guess one has to be careful to cache only variables that

only depend on parameters that are not updated in the current Gibbs inner iteration.

It seems that's not guaranteed by the approach you outlined above?

[mohamed82008]

I mean as a hack, I can expose @sampler() in TuringLang/Turing.jl#965 which lets the user access spl and do whatever he/she wants. Of course, the caching and guarantee that some computations only depend on some variables will be on the user.

[mohamed82008]

Done.

[mohamed82008]

In your code, you can do:

spl = @sampler()
if :x in spl.alg.space || :y in spl.alg.space
    # Compute var
    cache["var"] = var
else
    var = cache["var"]
end

[devmotion]

Out of curiosity, why is @sampler a macro?

[donhausk]

Thanks a lot for your comments!

[mohamed82008]

Out of curiosity, why is @sampler a macro?

Just because it is harder to overwrite a macro by mistake. This use case is rare enough that reserving a variable name for it inside the model body is a bit much imo. The macro is also defined in Turing and exported such that if used outside the model macro, it will error saying that this is only for use inside the model body.

[devmotion]

Ah I see 👍

Otherwise maybe one could define a macro @model_with_sampler that does basically the same as @model but also reserves the variable name sampler inside the model body. I'm not sure though if I would prefer that over a @sampler macro 🤷‍♂️

[yebai]

Closed in favour of TuringLang/AbstractPPL.jl#44