Expose lance metadata as tables

[kevinjqliu]

As discussed in today's "Lance Community Sync," I would like to propose exposing Lance metadata as tables/dataframes.

I found this approach quite helpful in the Iceberg community. As a data platform engineer (in my previous role), I frequently needed to quickly check the state of my tables. In Iceberg, this functionality is exposed as "metadata tables" (available in Spark, Trino, and PyIceberg) and includes information such as:

• History
• Metadata Log Entries
• Snapshots
• Entries
• Files
• Manifests
• Partitions
• Positional Deletes

A second order effect is that this enables analytics on table state, which can inform further optimizations when necessary.

Would love to hear if this would be helpful for others in the community!

[wjones127]

Yeah, I think this would be good.

A good starting place would be to choose a few of these to start with and define the schema we want for them. I'd suggest fragments (files can be a field in here), manifests, transactions, and indices might be the four first ones we want.

It would be cool if there was some basic projection and predicate pushdown supported eventually. For example, if you just select version and created_at from manifests, maybe we don't need to open each manifest file and instead can just use the list results. I would create an engine-agnostic API, and then you could bind it to a DataFusion TableProvider or Spark DataSourceV2 or similar.

[kevinjqliu]

Thanks Weston! I agree we should pick out a few and define a structure that can be easily replicated so others can add and extend. (Your database persona is already thinking about optimizations 😄😄😄)

ps there's an ongoing effort to add metadata tables in iceberg-rust (apache/iceberg-rust#2089)

[westonpace]

Thanks Weston!

Technically that was Will, but he does such great work I am always happy to take credit for it 😆

I think my only initial concern would be that the metadata tables should focus on the spec itself, and not extrapolate. For example, we could expose fragments as "partitions" in an effort to align with external APIs. However, there are other ways we could expose partitions (e.g. row ranges). If we pick one for a metadata table then we start to bake in assumptions that aren't formally part of the spec.

That being said, I don't think you are suggesting this, and I think this concern is just paranoia. Also, if a tool wants to then map that metadata table into whatever external format they want they are welcome to do so (e.g. SELECT fragment.id as partition_id FROM ...)

So I think this should be a pretty straightforward task of taking the protobuf and converting it to tabular format?

[kevinjqliu]

oops, sorry about that. I just saw the w 😄. Thank you both.

So I think this should be a pretty straightforward task of taking the protobuf and converting it to tabular format?

yep thats what im thinking too. The initial implementation should just expose the protobuf as is. We can get fancy afterwards if theres a need

[majin1102]

Hi, @kevinjqliu. Thanks for raising this discussion. Can't wait to see this metadata table in Lance.

I opened a discussion #5885 about wether to introduce a snapshot conept in Lance. I view this work might be a prerequisite for implementing a metadata table. This discussion primarily involves two aspects:

1. How to provide time travel on top of existing Lance while optimizing IO efficiency
2. Whether to combine the current Version and Transaction into a unified Snapshot concept to support a future snapshot table（IMO, snapshots metadata tables might need transaction-related metameta like operation and transaction_properties).

We'd love to hear your thoughts on this