Proposed format change: Add files to IndexMetadata

[wjones127]

Background

Currently, the table format doesn't know about any specific files that are part of an index. An index is given a directory, with it's UUID as the name, and it is expected to put all files in there.

It's up to the plugin to define how to open that index. To cleanup the index, we just list the directory and delete all objects within.

Proposal

We can add a message to the IndexMetadata:

message IndexMetadata {
    repeated IndexFile index_files = 10;
}

message IndexFile {
    string path = 1;
    int64 size = 2;
}

This gives us the exact size of the files within the index, which provides two benefits:

• Cold read performance: the index reader can skip the initial HEAD call to find the footer, if we store the size of the index file.
• Makes index size stats available: We could report the on-disk size of the index in statistics, without having to open the index.

[westonpace]

This seems like a reasonable idea to me. I suppose the primary downside is a very minor increase in the manifest size? Since index files are typically not per-fragment I think the increase would be fairly negligible.