Inducing point selectors

[henrymoss]

New functionality that allows us to dynamically fiddle with the inducing points used by our SparseVariational models as the optimization progresses. At the moment we only support using the same inducing points throughout (as specified when building the underlying gpflow SVGP model).

1. Added new top level class and 3 example subclasses (random, uniform and k-means) allowing the specification of inducing point selection routines. Note that this only currently works for models with a single set of inducing points, i.e. not for multi-output models with separate sets of inducing points for each output. However, this PR does support multi-output models with shared inducing points (which is more common that using separate sets). Its not clear how to update the points in multi inducing points case. I will have a think.
2. Fixed a bug that was stopping us from using these independent inducing point model at all (a dodgy assert in model.update). I have added in new unit tests to make sure this doesn't happen again. So to be clear, we support models with independent inducing points, we jsut dont have functionality to update their points as the optimization progresses.
3. By adding using one of these inducing point selectors in our integration test SVGP, I am able to greatly reduce the number of inducing points required for convergence and so have sped up the SVGP integration tests by an order of magnitude! This is good evidence that the inducing point selectors are super important!

[uri-granta]

Looks good! Just a few minor comments.

[uri-granta]

Is the bulk of this logic specific to SparseVariational or is it the same in all SupportsGetInducingVariables models? If so, should we perhaps update SupportsGetInducingVariables with an abstract set_inducing_variables(new_inducing_points, new_q_mu, new_q_sqrt) method, and a concrete update_inducing_variables(new_inducing_points) method which calculates the new q values and calls it?

[hstojic]

these are good questions, quite relevant as we need to move SGPR to its own wrapper...

[henrymoss]

I dont know how the SGPR works. In fact there are lots of issues with using it at the moment. Namely, its trajectory_sampler, conditonal_predict* methods, and covariance_between_points all are assuming that it is a GPR. I think we should get rid of it.

In summary, we never use it and none of the stuff thats meant to work with it does

[henrymoss]

I wont make these changes in this PR

[hstojic]

looks ok to me...

• I would move some code to utils.py
• the other major comment is to give it a think of how much of this can be reused in an SGPR wrapper, and then potentially move things so that the code can be easily reused in a future SGPR wrapper

[hstojic]

these are good questions, quite relevant as we need to move SGPR to its own wrapper...