Proposed format change: Add files to IndexMetadata

[wjones127]

Background

Currently, the table format doesn't know about any specific files that are part of an index. An index is given a directory, with it's UUID as the name, and it is expected to put all files in there.

It's up to the plugin to define how to open that index. To cleanup the index, we just list the directory and delete all objects within.

Proposal

We can add a message to the IndexMetadata:

message IndexMetadata {
    repeated IndexFile index_files = 10;
}

message IndexFile {
    // Path relative to UUID directory.
    string path = 1;
    int64 size = 2;
}

This gives us the exact size of the files within the index, which provides two benefits:

• Cold read performance: the index reader can skip the initial HEAD call to find the footer, if we store the size of the index file.
• Makes index size stats available: We could report the on-disk size of the index in statistics, without having to open the index.

Backwards compatibility

If index_files is empty, we can interpret this information as missing. Index readers can call HEAD as usual to get the size of files. Index stats can use a list operation as a fallback to get on-disk size of indices.

Index file paths

All index files are still required to be under a UUID directory. They can be at different base paths. So the full path of an index file would be in the format:

{Manifest.base_paths[IndexMetadata.baseId].path}/_indices/{IndexMetadata.uuid}/{IndexFile.path}

[westonpace]

This seems like a reasonable idea to me. I suppose the primary downside is a very minor increase in the manifest size? Since index files are typically not per-fragment I think the increase would be fairly negligible.

[jackye1995]

+1!

We recently added the base_id in Index metadata, so all the index files listed here will just inherit that base, I don't think we need per-file base yet and we an always add that later. https://github.com/lance-format/lance/blob/main/protos/table.proto#L282

[jackye1995]

One second thought @wjones127 , so far all the index file names are pre-defined as a part of the spec already. For example, we expect a file with name exactly index.idx, auxillary.idx, etc.

It is basically redundant to have the IndexFile repeating the file path. If the goal is to just have stats of the files for faster access, would it make more sense to just have additional stats fields in each index that knows what files are available?

[wjones127]

so far all the index file names are pre-defined as a part of the spec already.

I think we want to leave the door open for someone to implement a index format that isn't part of the core spec.

Part of this is separation of concerns.

That table format knows about IndexMetadata and knows how to parse and read it. The exception is the index_details field, which is opaque. It doesn't know the details of what files are in the index, what format they are in, or even what queries can be run on the index. To learn all that, it should call APIs on the index plugin.

The index plugin knows what files are part of the index, and how to read them and execute queries on them.

So, in line with your suggestion, we could implement an API:

impl IndexPlugin {
    fn file_sizes(&self) -> Vec<(Path, u64)> { ... }
}

That would make each index responsible for storing it's file size info somewhere it could quickly access. Ideally it would be stored in the index_details, since that requires no additional IO to retrieve.

This seems totally feasible, but I think needlessly complicates it. With this approach, each index now has to be responsible for storing and retrieving this data, even though there is just one optimal way to get it. By making the file list part of the table API, it enforces the ability to perform this lookup optimally.