What exactly constitutes the "Lance Format Specification"?

[westonpace]

The governance docs place stricter rules around changes to the specification but it is not clear what constitutes "the specification". Is it the protobufs? Is it instead the docs here https://lance.org/format/? Is any code included? Does lance-namespace have a specification? Does this include both the file and table specification?

It would be good to solidify this I think as it will help understand what needs special handling and what is just a regular change.

[wjones127]

At a minimum, I think any change to the protobufs are a format change. Is that controversial?

I think this covers both the file and table specifications.

[westonpace]

Another question is, "are index specifications part of the specification"? (e.g. #5485)

[majin1102]

That's a good question.

There's also a discussion around partition-per-table. For now we are assuming to provide partitions on namespaces and not sure to introduce any changes to the API. But some descriptions are already using specs to define this change.

Just share some of my thoughts: IMO A specification defines the fundamental structure and rules that multiple components and multiple adhere to.

1. Is it the protobufs?
   As Lance uses a lot of protobufs to define format structures. I think these protobufs are specs for Lance.

2. Is it instead the docs here https://lance.org/format/
   I think specs are always easy to document. Maybe we can treat https://lance.org/format/ as the basic guideline for specs.

3. Is any code included?
   In my view specs shouldn't include any code. Instead it replaces code with highly decoupled capability, so that components won't encounter compatibility issues due to code changes

4. Does lance-namespace have a specification?
   IMO this is a yes. namespace defines how to organize, find and manage metadata. It provides REST and other interfaces for multi-langs and engines to integrate. But I'm not convinced lance-namespace shoud belong to Lance format(IMO this spec could cover multi-format potencially)

5. Does this include both the file and table specification
   This is a yes to me.

I understand that there's a simple criterion to determine if a change belongs to the spec: in principle, a spec can only have additions and deprecations, not direct modifications. Directly modifying a spec will cause clear compatibility issues. However, we can introduce a new spec to gradually phase out the old one. This process is typically high-risk and time-consuming, which is why we must define our specs with great caution.(IMO sth like API of format, the real API could differ across langs and repos based on specs)

6. are index specifications part of the specification
   IMO this is a yes.