feat: support create vector index distributedly#5117

Merged

yanghua merged 72 commits intolance-format:mainfrom

chenghao-guo:ivf_distribute_builder

Jan 4, 2026

Commits on Jan 4, 2026

feat: distribute IVF assignment phase

chenghao-guo
and
yanghua
committed
fix: enforce global IVF/PQ training reuse in storage
chenghao-guo
committed
tests(vector): remove fallback; recall-only consistency across IVF/HNSW variants
chenghao-guo
committed
refactor: remove useless methods in index_merger

yanghua
authored and
chenghao-guo
committed
refactor: remove useless methods in index_merger

yanghua
authored and
chenghao-guo
committed
refactor: remove useless methods in index_merger

yanghua
authored and
chenghao-guo
committed
refactor: remove useless codes

yanghua
authored and
chenghao-guo
committed
fix: fix incorrect validation and fix style
chenghao-guo
committed
fix test issue

yanghua
authored and
chenghao-guo
committed
fix clippy issue

yanghua
authored and
chenghao-guo
committed
add test for index merger

yanghua
authored and
chenghao-guo
committed
add python e2e test

yanghua
authored and
chenghao-guo
committed
add python e2e test

yanghua
authored and
chenghao-guo
committed
add python e2e test

yanghua
authored and
chenghao-guo
committed
add python e2e test

yanghua
authored and
chenghao-guo
committed
add python e2e test: test_distributed_pq_order_invariance

yanghua
authored and
chenghao-guo
committed
add py test test_distributed_ivf_pq_order_invariance

yanghua
authored and
chenghao-guo
committed
try to refactor build_distributed_vector_index

yanghua
authored and
chenghao-guo
committed
refactor code

yanghua
authored and
chenghao-guo
committed
refactor code: remove useless code

yanghua
authored and
chenghao-guo
committed
refactor: comments of merge_index_metadata

yanghua
authored and
chenghao-guo
committed
refactor: remove duplicated code for create_index method

yanghua
authored and
chenghao-guo
committed
refactor: remove useless variable

yanghua
authored and
chenghao-guo
committed
refactor: test_vector_index.py

yanghua
authored and
chenghao-guo
committed
add test: test_empty_hnsw_fallback_matches_flat_search

yanghua
authored and
chenghao-guo
committed
add test: test_find_partitions_fallback_centroids_none

yanghua
authored and
chenghao-guo
committed
add test for ivf_sq, IVF_HNSW_SQ

yanghua
authored and
chenghao-guo
committed
add more tests

yanghua
authored and
chenghao-guo
committed
refactor import and use statement

yanghua
authored and
chenghao-guo
committed
add test : test_merge_ivf_pq_success

yanghua
authored and
chenghao-guo
committed
add more tests

yanghua
authored and
chenghao-guo
committed
refactor builder and merger

yanghua
authored and
chenghao-guo
committed
refactor: make prepare_global_ivfpq arg optional

yanghua
authored and
chenghao-guo
committed
refactor merge_partial_vector_auxiliary_files method name

yanghua
authored and
chenghao-guo
committed
refactor: introduce finalize_distributed_merge in rust

yanghua
authored and
chenghao-guo
committed
fix review suggestions

yanghua
authored and
chenghao-guo
committed
fix review suggestions

yanghua
authored and
chenghao-guo
committed
revert test case

yanghua
authored and
chenghao-guo
committed
fix review suggestions

yanghua
authored and
chenghao-guo
committed
reduce and remove some duplicated test cases

yanghua
authored and
chenghao-guo
committed
reduce and remove some duplicated test cases: test_distributed_ivf_sq_consistency, test_distributed_ann, test_distributed_flat

yanghua
authored and
chenghao-guo
committed
fix code style issue

yanghua
authored and
chenghao-guo
committed
reduce and remove generate centroids and pq_code_book

yanghua
authored and
chenghao-guo
committed
fix test issue

yanghua
authored and
chenghao-guo
committed
fix test issue and removed some validation logic

yanghua
authored and
chenghao-guo
committed
fix test issue

yanghua
authored and
chenghao-guo
committed
refactor code

yanghua
authored and
chenghao-guo
committed
refactor code

yanghua
authored and
chenghao-guo
committed
fix test issue

yanghua
authored and
chenghao-guo
committed
revert code

yanghua
authored and
chenghao-guo
committed
remove useless code

yanghua
authored and
chenghao-guo
committed
remove useless code

yanghua
authored and
chenghao-guo
committed
refactor partial dir naming pattern

yanghua
authored and
chenghao-guo
committed
try to fix merge order stable

yanghua
authored and
chenghao-guo
committed
try to fix merge order stable

yanghua
authored and
chenghao-guo
committed
refactor code

yanghua
authored and
chenghao-guo
committed
fix refactor code

yanghua
authored and
chenghao-guo
committed
remove hnsw related indices SupportedDistributedIndices

yanghua
authored and
chenghao-guo
committed
remove get_partial_pq_codebooks

yanghua
authored and
chenghao-guo
committed
remove get_pq_codebook

yanghua
authored and
chenghao-guo
committed
recover reader.rs

yanghua
authored and
chenghao-guo
committed
minor refactor

yanghua
authored and
chenghao-guo
committed
recover storage.rs

yanghua
authored and
chenghao-guo
committed
recover ivf/v2.rs

yanghua
authored and
chenghao-guo
committed
recover index.rs

yanghua
authored and
chenghao-guo
committed
recover index.rs and test_vector_index.py

yanghua
authored and
chenghao-guo
committed
refactor test code

yanghua
authored and
chenghao-guo
committed
refactor test code and distributed merger

yanghua
authored and
chenghao-guo
committed
refactor merger and builder

yanghua
authored and
chenghao-guo
committed
refactor vector.rs

yanghua
authored and
chenghao-guo
committed
address review comments
yanghua
committed
address review comments
yanghua
committed