Add a way to show the contents of the FileStatisticsCache in `datafusion-cli`

[alamb]

Is your feature request related to a problem or challenge?

• Related to Bug: statistics not collected automatically upon creation of ListingTable #18952

I am trying to understand if/when the FileStatisticsCache cache is used

However, now there is no way to understand the contents of the cache.

@nuno-faria made a really nice feature to view the contents of the cache: metadata_cache()

For example:

> select * from metadata_cache();
+---------------------------------------------------+---------------------+-----------------+--------------------------------------+---------+---------------------+------+------------------+
| path                                              | file_modified       | file_size_bytes | e_tag                                | version | metadata_size_bytes | hits | extra            |
+---------------------------------------------------+---------------------+-----------------+--------------------------------------+---------+---------------------+------+------------------+
| hits_compatible/athena_partitioned/hits_1.parquet | 2022-07-03T15:33:57 | 174965044       | "1f5da68e097309811a675c849491ac48-9" | NULL    | 165128              | 0    | page_index=false |
+---------------------------------------------------+---------------------+-----------------+--------------------------------------+---------+---------------------+------+------------------+
1 row(s) fetched.
Elapsed 0.005 seconds.

Describe the solution you'd like

I would like a table function similar to metadata_cache() for the statistics cache.

Someting like

select * from statistics_cache();

path	file_modified	file_size_bytes	e_tag	statistics_size_bytes
/foo/bar	2022-07-03T15:33:57	1234	...	132
/foo/baz	2022-07-03T15:33:57	5678	...	3112
...	...	...	...	...

Where statistics_size_bytes shows the size of the statistics, in bytes

Describe alternatives you've considered

No response

Additional context

No response

[nuno-faria]

@alamb I can take a stab at this. Just want to get some feedback first:

• The FileStatisticsCache will need to be a trait with a list_entries type function to get the cached entries, just like FileMetadataCache. Since FileMetadataCache is public, it will cause breaking changes, but it shouldn't be difficult to migrate.
• In your example there is a statistics_size_bytes, but I don't think that information is directly available in Statistics, unlike with the cache metadata entries. However, looking at the Statistics struct it appears to only store two Precision<usize> + five Precision<usize | ScalarValue> per column stored, which should be relatively small.

[alamb]

Thank you @nuno-faria 🙏

The FileStatisticsCache will need to be a trait with a list_entries type function to get the cached entries, just like FileMetadataCache. Since FileMetadataCache is public, it will cause breaking changes, but it shouldn't be difficult to migrate.

I agree we'll need a real trait and your proposal makes sense to me

In your example there is a statistics_size_bytes, but I don't think that information is directly available in Statistics, unlike with the cache metadata entries. However, looking at the Statistics struct it appears to only store two Precision + five Precision<usize | ScalarValue> per column stored, which should be relatively small.

I think this can get potentially quite large for wide tables or with large numbers of columns. For example, a ScalarValue is 64 bytes each, so that means a 20 column table with 10 row groups each takes 6KB (without accounting for the size of Vecs, etc): 16 + (20 * 5 * 64)

I recommend:

1. For the first PR, report a memory size of -1 or something to show it isn't actually accurate, and limit the size of the cache by number of Statistics objects
2. As a follow on PR, we can add a way to calculate the memory usage and also limit the memory usage by size

[alamb]

I looked at the code a bit more, and I think the current setup means there is no global statistics cache (by default each table provider gets its own individual cache).

I filed a ticket to track limiting the cache size and making one session-level cache here (basically following your example from the statistics cache):

• Add limit to DefaultFileStatisticsCache #19052

I suspect once we did that then showing the contents of the cache will be much simpler

[alchemist51]

one session-level cache here

Didn't understand this fully 😅 , we are creating the Statistics cache in the CacheManager right which is a Session object. How is it still constraint to a table?

[alamb]

one session-level cache here

Didn't understand this fully 😅 , we are creating the Statistics cache in the CacheManager right which is a Session object. How is it still constraint to a table?

Somehow it seems like this is the statistics cache that gets created (not connected to the session)

https://github.com/apache/datafusion/blob/9f725d9c7064813cda0de0f87d115354b68d76e6/datafusion/catalog-listing/src/table.rs#L227-L226

So making a PR to show the statistics cache is great, but I suspect it won't show up for a while