Rewrite to remove normalization constants from joint_logprob

[ricardoV94]

There was some work documented here already: pymc-devs/pymc#4396

Would be a great feature to include in the library.

CC'ing @manthehunted on the off-chance that he/she is still interested in this

[manthehunted]

@ricardoV94

Yes I am. I am just not sure what else I need to do. Since I do not use pymc3 or Bayesian framework as much, I am not sure what are completion criteria.

It will be great if someone can specify the end goal, other than "we like to do this" (because a normalization factor for a simple case is done). Then I am totally happy to push this to the end.

[ricardoV94]

It will be great if someone can specify the end goal

I guess we need to check if any obvious optimizations are missed by your implementation when using realistic models (e.g., hierarchical models, variable transformations, mixture models, etc), examples of which can be found in the current tests.

And if things are working alright, we need tests. Should be as simple as evaluating a couple of models with and without the optimization and checking that indeed they differ only by a constant term.

@brandonwillard might have more concrete advice here.

[brandonwillard]

Before I forget, here's a Gist with the beginnings of an implementation for all this.

It has the added constraint that it doesn't want to change the multiplication order of terms, so it "uniquifies" terms in a graph in order to keep track of exactly which terms were being multiplied amid distributive expansions.

Without that rather strong requirement, something like RemoveMulConstants alone would do the job.

[twiecki]

@brandonwillard Anything from @manthehunted work https://gist.github.com/manthehunted/66f0560384b7456ce831c853535ab796#file-normalize_v2-ipynb that we can reuse here?