Multi-phase, prolonged feature development discussion.

[numinnex]

As we are approaching the end of development of the io_uring_tpc feature, I'd like to propose a different development framework for the next (even bigger) feature, clustering.

Background: Current Approach

For io_uring_tpc we used a long-lived feature branch, with no CI checks during merges (so we could merge changes incrementally from other contributors, without needing to fix the tests a priori) and branch protection enabled (1 approval).

This initially sounded like a reasonable approach, with a few downsides (e.g., contributions to master that touch server had to be frozen). However, as development continued, it became clearer that this approach is problematic.

Pain Points

• Due to branch protection being enabled, it made it impossible to rebase it against master, as it requires pushing with --force.
• Since rebase was not an option, we were forced to merge the master branch into the feature branch, which in the long run led to commit duplication (some of the commits on the feature branch from previous upstream merges were conflicting with the same commits that were brought by the current merge).
• The assumption that we have to freeze the master branch from changes to the server was foolish. In reality, you can't predict what parts of the codebase will require changes when developing a feature.
• Introducing external contributors to the workflow introduces extra obstacles (different workflow compared to master branch, created issues require a special label that directs towards the feature branch).

Proposed Framework

Using those lessons, I propose a different development framework. Deprecate the idea of a feature branch as the core development branch, use master branch instead. In order to avoid conflicts between fixes/patches that would usually go directly to master, the feature currently under development must be divided into the smallest possible atomic units, such that each of those units becomes a separate crate. Each of those units is assigned to one of the contributors (single responsibility).

Contributors can upstream small changes frequently without conflicts, allowing us to move fast and identify interface incompatibilities during code review.

There is no need for a master branch freeze anymore, as there should be no contention between the feature development and fixes/patches/other contributions. The only time when a branch freeze is required is during the "plumbing" (combining the modules together).

Additional Benefits

Another upside of this approach is moving towards our goal of modularizing the codebase, we are killing two birds with one stone.

Addressing Concerns

At first glance it looks like a lot of work to get into a workable solution (e.g., pre-planning the feature division into crates), but I'd argue that:

• We need to do it anyway (goal of modularizing).
• This won't be the last time we'll need to work on a feature for an extended period.

So I deem the time investment worth it.

Discussion

If you have any other experiences working with such scenarios, please share them in this discussion. From my side, I've worked with long-lived feature branches twice, both times it turned out to be painful.