Index optimize extension with dynamic index support in embed db

[coyzeng]

In my case, my mysql server support multi different table, like API table, private table etc..., some table index must be dynamic for support index optimize with complex condition.

[max-hoffman]

Hi @coyzeng, thank you for taking the time to contribute this interface suggestion. We have also been spinning with some ideas of how to sharpen up the indexed access interface.

The current way we support this is by having an indexable table source GetIndexes() return a variety of indexes depending on the specific version of the table pinned to the session, even if those index expressions will not be used in a predictable way from GMS's standpoint. Would be willing to share more about your use case? Is your table indexable on any set of columns, but has many hundreds of columns so an exhaustive list of indexes subsets is impractical to pass through the GetIndexes() interface?

[coyzeng]

Hi @coyzeng, thank you for taking the time to contribute this interface suggestion. We have also been spinning with some ideas of how to sharpen up the indexed access interface.

The current way we support this is by having an indexable table source GetIndexes() return a variety of indexes depending on the specific version of the table pinned to the session, even if those index expressions will not be used in a predictable way from GMS's standpoint. Would be willing to share more about your use case? Is your table indexable on any set of columns, but has many hundreds of columns so an exhaustive list of indexes subsets is impractical to pass through the GetIndexes() interface?

I have some logic table, they work with dynamic index, like virtual index. If query condition can not hit the fixed index, it will scan all rows of the table. Dynamic index work without scan all rows of the table.

[zachmu]

The interceptor chain is interesting and could be useful, but I have a couple suggestions on it.

The index change I don't think is necessary. Take a look at my comments and see if they make sense.

[zachmu]

Name the receiver ci to match package style

[coyzeng]

Done

[zachmu]

I don't think you want or need a new interface to accomplish your goals here. If you want to implement a custom index type that matches some subset of expressions, your best bet is for your table implementation to return N indexes, one for each combination of expressions you are capable of indexing. This is how Dolt originally implemented index prefix matching before we implemented it natively in go-mysql-server.

[coyzeng]

It's difficult to match any index without query condition, 'IndexAddressable' not support query condition in go-mysql-server now. This solution is the simplest way to implement some index feature without code changes. Virtual table need dynamic index to skip scan all rows of table.

[coyzeng]

If use'IndexAddressable' for table implementation, it need build too many complex index to match query. The number of columns combinations is very big and complex.

[max-hoffman]

One thing we could maybe support is a new interface that does something like this:

type IndexSearchable interface {
    IndexWithPrefix([]string) []sql.Index
}

You would switch for this type before falling back to the old expression checking pattern.

We are going to delete indexAnalyzer and all of the existing index matching code shortly, to be replaced with a very different way of representing indexes and expressions. New index matching interfaces needs to be compatible with how this logic will work in a couple months

[zachmu]

Max's suggestion sounds reasonable. The rest of the PR looks reasonable if you make that change.

Thanks for the contribution!

[zachmu]

This should be a configuration option on cfg, not hardcoded here. We don't want the interceptor chain installed by default (even with an empty chain)

[coyzeng]

Done

[zachmu]

Changes look good except for the new interface. Implement Max's suggestion and we'll merge it in.

[zachmu]

Max's suggestion sounds reasonable. The rest of the PR looks reasonable if you make that change.

Thanks for the contribution!

[coyzeng]

Changes look good except for the new interface. Implement Max's suggestion and we'll merge it in.

I have implemented Max's suggestion. But I'm not sure it's the best way. Max says indexAnalyzer will be remove shortly.

I also want to create another PR for mysql fully homomorphic encryption pluggable expression support. Which make go-mysql can work on multi party data with privacy computing.

[max-hoffman]

This is what the index scan rewrite will look like, as a reference #2093. It will take 4-5 more PRs for this to replace the current form, but I've thought about it more thoroughly now. The interface you're suggesting would fork the implementation based on whether you want a costed approach to choosing indexes, or defer to the database to choose its own preferred index. All I need is the hook for 1) checking if the database wants to override statistical index checking, and 2) a method that gives the table a list of equality filters, and returns a sql.Index or sql.IndexLookup in return. So a slight edit to my previous suggestion.

type IndexSearchable interface {
    SkipIndexCosting() bool
    IndexWithPrefix([]string) sql.IndexLookup
}

Supporting range filters might need a different interface.

Just trying to make sure these two PRs converge in a way that doesn't break compatibility for you.

[coyzeng]

This is what the index scan rewrite will look like, as a reference #2093. It will take 4-5 more PRs for this to replace the current form, but I've thought about it more thoroughly now. The interface you're suggesting would fork the implementation based on whether you want a costed approach to choosing indexes, or defer to the database to choose its own preferred index. All I need is the hook for 1) checking if the database wants to override statistical index checking, and 2) a method that gives the table a list of equality filters, and returns a sql.Index or sql.IndexLookup in return. So a slight edit to my previous suggestion.

type IndexSearchable interface {
    SkipIndexCosting() bool
    IndexWithPrefix([]string) sql.IndexLookup
}

Supporting range filters might need a different interface.

Just trying to make sure these two PRs converge in a way that doesn't break compatibility for you.

I redeclare IndexSearchable as below and make a slight edit. This is the minimal edit before #2093 draft merged.

type IndexSearchable interface {
	// SkipIndexCosting will choose the database preferred index if return true.
	SkipIndexCosting() bool
	// IndexWithPrefix returns an array of this table's Indexes that matched with prefix.
	IndexWithPrefix(ctx *Context, expressions []string) (Index, error)
	// IndexWithExpression returns an array of this table's Indexes that matched with expression.
	IndexWithExpression(ctx *Context, expressions []Expression) (Index, error)
}

[max-hoffman]

I finished up the indexScan rewrite today, and I ported your interface suggestion here #2125. Can you take a look and tell me whether this works for you? Reusing engine logic and implementing IndexWithPrefix is an optional extension of this, I wasn't sure whether you wanted that right now or just the bare bones version.

You could edit and send back a different PR if you need more functionality. Either way, it might be easier to separate the interceptor chain and index changes into two different PRs.