Add a new row count estimation mechanism for CalciteIndexScan

[qianheng-aws]

Description

Add a new row count mechanism for CalciteIndexScan, which is a new TableScan implemented by us for opensearch index.

Before this PR, this operator reused the same cost estimation logic as its parent class TableScan, which provided a constant value for its cost (i.e., 100 rows). However, since we also support pushing down operators like Project, Filter, Aggregation, etc. into this scan operator, and some operators actually affect the output rows of scan, we need a new implementation for this scan operator. Otherwise, in some cases, the Calcite optimizer will pick incorrect plans that don't actually have the lowest cost.

To estimate the cost of CalciteIndexScan more precisely, we need to imitate the row estimation logic implemented in RelMdRowCount. To make the Calcite optimizer prefer plans with push down rather than without, we add a factor (0.9) multiplied to the original row count of scan.

Background

• In Calcite, by default, the costs of an operator are estimated completely based on their row count, and it will never let the result of row count goes below 1.

Related Issues

Resolves
#3557
#3556

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[penghuo]

For scan operator, It should be index docs count? the query.size_limit configuration sets the maximum number of rows that a query can return, but not the index scan.

[qianheng-aws]

Yeah, to be more precisely, it should be docs count.

But it still works fine here to give a constant value here, because:

1. By default behavior without this PR, Calcite will also provide a constant value 100
2. We only need a relative value instead of a precise value for the cost computing, especially for the single table scanning. In the cases of multi-table joining, it should make difference if we introduce optimization like join-reorder, map-join, but seems those optimization only make sense in distribution environment.
3. Using the real doc count will introduce one more request but won't have much worth as said in 2.

[penghuo]

Can we use a fixed value (e.g., 10K) or a different setting? Using query.size_limit, which is unrelated to cost estimation, can be misleading. We don't expect changes to query.size_limit to affect the query pushdown logic, correct?

[LantaoJin]

[question] why need this flag? what case will the scan be estimated multiple times? and how many times? Is it possible to remove the member variable rowCount in class? The method estimateRowCount should work independently.

[qianheng-aws]

The estimateRowCount will be called when:

1. Computing the cost of this operator itself
2. Computing the cost of its parent operator and the computing logic of that is based on the input row count. The times depends on how many parent operators will be generated during the optimization process. For example, when having plan project-agg(avg)->scan, the parent operators of scan will at least includes agg(avg), agg(sum/count), EnumAgg and may be project if have optimization rule of pushing project through agg.

[LantaoJin]

Ok, keep this flag would be fine. But is it possible to remove the member variable rowCount in class? The method estimateRowCount() should work independently.

[qianheng-aws]

rowCount and rowCountEstimated is viewed as kind of cache for the final estimated row count.

It's OK to remove this cache since other operators don't have such cache.

[penghuo]

Update doc. https://github.com/opensearch-project/sql/blob/main/docs/user/admin/settings.rst