Statistic: data_size should be in ColumnStatistics.

[jackwener]

Is your feature request related to a problem or challenge?

No response

Describe the solution you'd like

Current, we use total_byte_size to store whole byte_size.

But it isn't good enough, a better way is to put avg_data_size/total_date_size into ColumnStatistics.

a total_byte_size of statistic is useless, because we hard to propagate it in Statistic derive.
But if we use Column avg_data_size we can use it to propagate it into other Plan.

Spark:

case class ColumnStat(
    ....
    avgLen: Option[Long] = None,
    maxLen: Option[Long] = None,
    ...

Presto

public final class ColumnStatistics
{
    private final Estimate nullsFraction;
    private final Estimate distinctValuesCount;
    private final Estimate dataSize;
    private final Optional<DoubleRange> range;
}

Describe alternatives you've considered

No response

Additional context

No response

[AdamGS]

I know this is a pretty old issue, but I would also be interested in it and capable of doing the implementation if there's interest by the maintainers.

[findepi]

@AdamGS +1 from me.
The average data size sounds most logical from optimizer's perspective
(i was involved in the introduction of ColumnStatistics.dataSize of Presto/Trino, but the reality is the value is so often divided by #rows ...)