Migrate existing unstable dataset to stable dataset

[yanghua]

Motivation and Goals

Today, stable row IDs in Lance must effectively be decided at dataset creation time. Once a dataset has been written without stable row IDs, there is no supported way to “upgrade” it in place so that future features(e.g., CDF and row lineage) can rely on stable row IDs.

This proposal introduces a metadata-only migration so that we can migrate an existing unstable dataset to a stable dataset. There would be some major steps:

• Takes an existing dataset that was written without stable row IDs;
• Constructs row-id metadata (RowIdSequence) for every existing fragment.
• Produces a new manifest version that:
  • Turns on FLAG_STABLE_ROW_IDS in the feature flags.
  • Populates row_id_meta for all fragments.
  • Sets a correct next_row_id value so future writes can continue allocating stable row IDs.
• Leaves the underlying data files unchanged; only metadata and optional row-id sidecar files are written.
• From that new version onward, the dataset behaves as if it had been created with enable_stable_row_ids = true from the beginning.

Key Constraints

• All fragments must have row-id metadata

Once we set FLAG_STABLE_ROW_IDS, the manifest invariants still apply: either all fragments have row_id_meta or none do. The migration must ensure that every fragment gets a valid RowIdSequence before flipping the flag.

• No in-place partial upgrade

The upgrade is done by creating a new manifest version. Older versions remain as-is (no stable row IDs); only the new version and onward use stable row IDs. There is no mixing of “some fragments with row IDs, some without” within a single manifest.

• Metadata-only operation

The migration must not rewrite data files. It only:

1. Writes per-fragment row-id sequences (inline or external).
2. Writes a new manifest that references these sequences and updates feature flags / next_row_id.

• Concurrency / write freeze

To avoid inconsistent views, the migration assumes: There are no concurrent operations while it’s running.

• Deletions are preserved

Existing deletion files remain untouched. The migration ensures that the resulting RowIdIndex still maps only live rows, consistent with current deletion semantics.

• Global row-id monotonicity

Logical row IDs assigned during migration:

1. Are **globally unique** across the dataset.
2. Are monotonically increasing (e.g., a global counter that grows across fragments).
3. `next_row_id1 is set to `max_assigned + 1` so future writes never reuse IDs.

[yanghua]

cc @westonpace @wjones127 @jackye1995 Just raised the idea and some major steps. Looking forward to here your feedback.

After that, I will draft the api design...

[wjones127]

• Concurrency / write freeze

To avoid inconsistent views, the migration assumes: There are no concurrent operations while it’s running.

My only issue is this. I think we should put the effort into making the transaction safe. This should be pretty easy actually.

1. Use the Overwrite operation to rewrite the fragments with their RowIDSequence
2. Change apply_feature_flags to set the stable row id flag when there are RodIdSequences in the fragments. This is already done:
   

   lance/rust/lance-table/src/feature_flags.rs

   Lines 48 to 66 in 135d0b9

   	// If any fragment has row ids, they must all have row ids.
   	let has_row_ids = manifest
   	.fragments
   	.iter()
   	.any(|frag| frag.row_id_meta.is_some());
   	if has_row_ids || enable_stable_row_id {
   	if !manifest
   	.fragments
   	.iter()
   	.all(|frag| frag.row_id_meta.is_some())
   	{
   	return Err(Error::invalid_input(
   	"All fragments must have row ids",
   	location!(),
   	));
   	}
   	manifest.reader_feature_flags |= FLAG_STABLE_ROW_IDS;
   	manifest.writer_feature_flags |= FLAG_STABLE_ROW_IDS;
   	}

That should give you basic conflict resolution for this and prevent inconsistent state.

The disadvantage of using Overwrite is that if there is a concurrent Append operation that commits first, then this will drop the new data. If you want to prevent this, then you could write a new operation with different conflict resolution rules.

[majin1102]

Just to catch up quickly, I’m wondering what issues we might encounter if we use Overwrite directly and enable stable_row_id. cc @yanghua

[zhangyue19921010]

Does using overwrite directly require rewriting the entire dataset or just using overwrite commit? If rewriting, it might be a little heavy and the index may also become invalid. Is that possible to do something like meta data only compaction/overwrite

Another point to note is that if only metadata operations are performed, it may not be possible to ensure the global uniqueness of Fragment IDs, because new commits will reuse all Fragment IDs from the previous commit.