feat: support merge_insert with source dedupe on first seen value

[jackye1995]

Based on feedback in #5582

Simplified implementation that just keep the first value in case of duplicated on rows in source during merge insert. Users are expected to sort the source properly before using merge-insert.

[github-actions]

Code Review

Summary: This PR adds a DedupeBehavior option to merge_insert that allows keeping the first-seen value when duplicate source rows match the same target row, instead of failing.

P1: Performance Issue in Duplicate Filtering

In merge_insert.rs around line 2075-2082, the duplicate filtering uses:

let keep_mask: BooleanArray = (0..matched.num_rows())
    .map(|i| Some(!duplicate_indices.contains(&i)))
    .collect();

This is O(n*m) where n is the batch size and m is the number of duplicates. For batches with many duplicates, this becomes expensive. Consider using a HashSet for O(1) lookups:

let duplicate_set: std::collections::HashSet<_> = duplicate_indices.iter().collect();
let keep_mask: BooleanArray = (0..matched.num_rows())
    .map(|i| Some(!duplicate_set.contains(&i)))
    .collect();

Other Observations

1. Test Coverage: Good parameterized test covering multiple configurations (full schema, stable row IDs, file versions).

2. API Design: The builder pattern with dedupe_behavior() follows existing conventions well.

3. Stats Tracking: The num_skipped_duplicates metric is properly propagated through both code paths (full-schema and non-full-schema merge insert).

Overall this is a straightforward and well-tested feature addition. The main actionable item is the performance concern above.

[codecov]

Codecov Report

❌ Patch coverage is 90.76087% with 17 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/write/merge_insert.rs	86.50%	15 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[wjones127]

Looks good!

[wjones127]

It would be more efficient to just track keep_indices, then later you can directly call take on the record batch with those indices. Then you don't need to construct the hashmap or boolean mask.