Datastore: Immediately create/drop indices in commited state and roll back if needed #2301
Description
Currently, when indices are created in the tx state, we end up putting committed state rows in the tx index.
This forces us to do a bunch of unnecessary checks in hot code, whereas index creation in a transaction is, relatively speaking, incredibly rare.
It would be better to pay the cost in migrations instead and immediately change the committed state (but not in committed system tables),
and keep track of these changes, such that if the migration failed, we can roll back by removing the created indices and putting back the dropped ones. Dropping an index and putting it back should not require building anything more than we do today. We can keep dropped indices around in the tx state until we truly know that we don't need it anymore.
These changes should help both simplify the code as well as improve performance.