What is the process for making a specification change?

[westonpace]

I propose the following process:

1. Early in the design phase a user should start a Github discussion to gather feedback on the proposal. This is to help avoid doing a lot of work that is then wasted because the community doesn't want a feature or wants to handle it in a different way. There might not be much feedback on the discussion, this is ok and should be interpreted as agreement.
2. A PR should be made the focuses on the specification change itself. This helps make it easier for people to understand what is beign voted on.
3. An additional PR implementing the change should be made. If the change requires compute engine support then the PR (or a separate PR) should include support for at least one engine (Datafusion, Spark, ...)
4. Once all these PRs are available then a vote discussion should be created. The discussion itself is mainly to draw attention to and highlight the PR. Users that vote +1 should also approve the PR.
5. Once sufficient time has passed and sufficient votes are achieved then the spec change PR should merge. The remaining PRs can merge at their leisure.

This is hopefully not a surprising process and is based on Arrow's guidelines. The only difference is that Arrow requires two implementations prior to a vote. However, I'm not sure there is an equivalent for Lance since it is not really meant to be an inter-language exchange protocol like Arrow is. We do have multiple engines but that's not quite the same thing and requiring two engines to implement a feature is premature at this point in time.

[HaochengLIU]

I like the process and personally I prefer one implementation. Get one ball moving in the right direction is good enough, given Lance is still fast evolving.

There is one more follow-up question: For each area, should we assign some moderators, or we just rely on the existing PMC members(minimal 1 vote) and maintainers (minimal 2 votes)? Currently there are quite some open issues and PRs, I feel we need to do some housekeeping in the nearing future and come up with a routine to handle them :)

[wjones127]

For each area, should we assign some moderators, or we just rely on the existing PMC members(minimal 1 vote) and maintainers (minimal 2 votes)?

This is a good question. It might be a good idea to actually create a CODEOWNERS.md file for the specification protos and other files. This would let us assign certain PMC members to certain parts of the spec. For example, we might have some looking at file format changes, some at table format changes, and some at namespace spec changes. Having codeowners might also help prevent spec changes sneaking through without a vote.

For areas, I think file format, table format, and namespace spec seems like an appropriate way to split things. What do you think of that?

[HaochengLIU]

^^
+100
I strongly support the codebaseowner.md as it works well according to the world of mouse around me

• For areas, I think file format, table format, and namespace spec seems like an appropriate way to split things. What do you think of that?

SGTM. One QQ: for indexes, which category should it fall into?

[wjones127]

for indexes, which category should it fall into?

I think Table format. They could be their own category but I think it makes sense to include them in Table format.

[wjones127]

This seems good.

Should a voting thread be in a brand new discussion? Or can it just be a new thread in the original design proposal discussion? I like the simplicity of the latter, but maybe there are downsides.

[zhangyue19921010]

+1 to this propose especially the two points:

• Discussion first: surface any directional disagreements early so we don’t end up doing a bunch of work only to find out the community doesn’t want the feature (or wants it done differently).
• Separating the Spec PR from the implementation PR: during the vote, reviewers can focus on the spec’s semantics, compatibility, and edge cases, rather than getting bogged down in implementation details.

I have two small questions:

1. Should we define a clear time window for both discussions and votes (e.g., 7 days)?
2. For the vote phase, if someone votes -1 (hopefully I won’t run into this 😄), how should we handle it? Do we need to pause the vote and block merging until that -1 changes to a +1?

[wjones127]

1. Should we define a clear time window for both discussions and votes (e.g., 7 days)?

If you look at https://lance.org/community/voting/#voting-requirements, there is already a 7 day requirement for votes.

I don't think we need to prescribe a time window for discussions, but we might give a suggested one. I think we should state that if someone doesn't give enough time for discussion and there are open questions remaining, that's a valid reason for a -1 vote. Therefore, contributors should have incentive to give an appropriate amount of time to discussion.

2. For the vote phase, if someone votes -1 (hopefully I won’t run into this 😄), how should we handle it? Do we need to pause the vote and block merging until that -1 changes to a +1?

This is also discussed in the voting docs. Basically, you are required to discuss in the voting thread and see if you can address the concerns from the -1 vote:

A -1 binding vote is considered a veto for all decision types. Vetoes:

• Stop the proposal until the concerns are resolved
• Cannot be overruled
• Trigger consensus gathering to address concerns

It does say explicitly that if someone gives a -1 vote without a justification, it can be ignored:

-1 votes must include justification to allow meaningful discussion. Any -1 vote not accompanied by justification is considered invalid.