Issue 6544 - Query performance improvements on large parquet files

[m09526]

Make sure you have checked all steps below.

Issue

• My PR fully resolves the following issues. I've referenced an issue in the PR title, for example "Issue 1234 - My
  Feature". Note that before an issue is finished, you can still make a pull request by raising a separate issue
  for your progress.
  • Resolves Query performance improvements on large Parquet files #6544

Tests

• My PR adds the following tests based on our test strategy OR does not need testing for this extremely good reason:
  • rust/sleeper_core/src/datafusion/util.rs

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it, or I have linked to a
  separate issue for that below.
• If I have added new Java code, I have added Javadoc that explains it following our conventions and style.
• If I have added or removed any dependencies from the project, I have updated the NOTICES file.

[patchwork01]

There are failing tests and Checkstyle problems.

[patchwork01]

The issue states that disabling reading page indexes in DataFusion isn't possible with the cached reader.
It looks like you've worked around this in rust/sleeper_core/src/datafusion/metadata_cache.rs?

Please can you explain in a comment here how these two things relate to one another, and what was intended?

Is there a way we can bring the code for this together into one place? It makes it a bit difficult to read with the read_page_indexes setting checked in multiple places.

[m09526]

Added more comments to code.

[patchwork01]

That helps, but I think the important part is the relationship between this line, the other check of read_page_indexes below, and the code in rust/sleeper_core/src/datafusion/metadata_cache.rs. From reading the comment here you wouldn't know that we have worked around the problem with DataFusion, or that those other pieces of code exist.