Improve JS API's usage of subscriptions #188
Description
To expedite development, JS is still using flatten, and re-creating subscriptions rather than updating them out-of-band. Fortunately, by flattening we reduce the cost of re-creating subscriptions, at least in terms of sending large sparse indexes.
The first fix should be to change both types of subscriptions to share the same code to allow for non-flat indexes - to achieve this, viewports will need to handle receiving a snapshot that only covers part of the new viewport. Ideally, out-of-band subscription updates should be implemented before this work is done.
Additionally, subscriptions are using ColumnData rather than just creating the real data that is used by the API - some of this indirection can be cleaned up. Doing this work will also let us make the kinds of data we can read much more flexible, and possibly consider delegating to arrow.
Breaking this down as bulletpoints rather than text above:
- Stop re-creating subscriptions when the viewport or columns change, but still subscribed to the same table
- Support the "reverse viewport" feature
- Stop requiring "flat" tables - some code should be sharable with the non-viewport subscription to achieve this, but it will need to be generalized to support viewports.
- Drop the ColumnData, TableSnapshot, DeltaUpdates intermediate types, as long as so much of the above is being rewritten, to make it easier to support new types.