[QDP] Initiate project QDP #786

guan404ming · 2026-01-04T17:49:45Z

Purpose of PR

Initiate QDP project

Related Issues or PRs

Changes Made

Breaking Changes

Yes
No

Checklist

Added or updated unit tests for all changes
Added or updated documentation for all changes
Successfully built and ran all unit tests or manual tests locally
PR title follows "MAHOUT-XXX: Brief Description" format (if related to an issue)
Code follows ASF guidelines

* structure * fix a lots of cuda errors * change folder name * improve

* Add Python bindings for QDP using PyO3 * Add QuantumTensor class for DLPack integration * Add TODO * Update qdp/qdp-python/src/lib.rs Co-authored-by: Ryan Huang <[email protected]> --------- Co-authored-by: Ryan Huang <[email protected]>

* add nvtx and some comments * Update qdp/docs/observability/NVTX_USAGE.md Co-authored-by: Ryan Huang <[email protected]> * catch cuda include path * fix prefix errors * fix precommit error and rebuild Cargo.lock * fix precommit error * fix precommit errors --------- Co-authored-by: Ryan Huang <[email protected]>

* add preprocessing module * Remove unused import of Cow

Signed-off-by: 400Ping <[email protected]>

* async pipline * prevent error * Add TODO for dynamic chunk size tuning Added a TODO comment to tune chunk size dynamically based on GPU model or PCIe bandwidth. * update * add a test

* Integrate Apache Arrow & Parquet for data processing * Optimize Arrow Float64Array handling in io * Add chunked Arrow Float64Array support * Refactor encoding to support chunked Arrow Float64Array input * Refactor I/O and encoding documentation to remove zero-copy

* add vanilla gpu kernel Signed-off-by: ryankert01 <[email protected]> * Optimize amplitude encoding kernel with safety checks Refactored kernel to use multiplication for normalization and optimized memory access with double2. Added safety checks for input parameters. * Remove comments on norm validation and optimization Removed comments regarding norm checks and optimization. * add docker dev container and test Signed-off-by: ryankert01 <[email protected]> * add docker dev containe Signed-off-by: ryankert01 <[email protected]> * delete unrelated strings Signed-off-by: ryankert01 <[email protected]> * delete unrelated strings Signed-off-by: ryankert01 <[email protected]> --------- Signed-off-by: ryankert01 <[email protected]>

* Batch Throughput Signed-off-by: 400Ping <[email protected]> * fix Signed-off-by: 400Ping <[email protected]> * [chore] add comment Signed-off-by: 400Ping <[email protected]> --------- Signed-off-by: 400Ping <[email protected]>

* gracefully handles OOM Signed-off-by: 400Ping <[email protected]> * fix Signed-off-by: 400Ping <[email protected]> --------- Signed-off-by: 400Ping <[email protected]>

* add a fidelity test * fix format * reformat * effectively zero

* improve amplitudeEncoders for less copy memory allocations * Fix: apply pre-commit end-of-file-fixer * use pre-processing module

Updated build instructions to include Python 3.11 environment setup.

) * [QDP] [Benchmark] Add a End-to-End benchmark to compare with speed * [QDP] improve memory management * Revert "[QDP] improve memory management" This reverts commit d909f3d.

* [QDP] Add batch encoding support * Refactor batch pre-processing

* Add Arrow IPC support * Update default frameworks to "mahout-parquet" * Add tests for Arrow IPC * Add todo to prevent oom * Add TODO for supporting multiple float types

Co-Authored-By: Ping <[email protected]>

* GPU/Cuda kernel Optimizations Signed-off-by: 400Ping <[email protected]> * fix Signed-off-by: 400Ping <[email protected]> --------- Signed-off-by: 400Ping <[email protected]>

* improve e2e benchmark * gc.coolect first

* Set float32 to default Signed-off-by: 400Ping <[email protected]> * fix errors Signed-off-by: 400Ping <[email protected]> * fix python bindings Signed-off-by: 400Ping <[email protected]> * [chore] update log Signed-off-by: 400Ping <[email protected]> --------- Signed-off-by: 400Ping <[email protected]>

Signed-off-by: 400Ping <[email protected]>

* DataLoader-Benchmark Signed-off-by: 400Ping <[email protected]> * update output Signed-off-by: 400Ping <[email protected]> * update Signed-off-by: 400Ping <[email protected]> * [Fix] Remove print in Qiskit Signed-off-by: 400Ping <[email protected]> * fix Signed-off-by: 400Ping <[email protected]> * fix pre-commit Signed-off-by: 400Ping <[email protected]> * [QDP] improve memory management * fix pre-commit Signed-off-by: 400Ping <[email protected]> * update doc Signed-off-by: 400Ping <[email protected]> * update doc Signed-off-by: 400Ping <[email protected]> * improve memory usage * improve pool * improve for memory * fix back * merge and improve * fix error * memory release * follow suggestions * Address reviewer feedback: safety checks, error handling, and code cleanup * Revert e2e Benchmark * refactor ffi * remove redundant check * fix error --------- Signed-off-by: 400Ping <[email protected]> Co-authored-by: 400Ping <[email protected]>

* docs: add development guide * refactor: precommit * docs: add dev container in DEVELOPMENT * docs: add more troubleshooting

* [QDP] Fix DLPack device_id hardcoding# Please enter the commit message for your changes. Lines starting * update and move comments position * improve the test

* float32 follow up Signed-off-by: 400Ping <[email protected]> * update Signed-off-by: 400Ping <[email protected]> --------- Signed-off-by: 400Ping <[email protected]>

* refactor: benchmark/ move into qdp-python/ * build: add benchmark group in pyproject.toml * build+docs: add make commands and update development docs * build+docs: add profiling make command and docs * fix: unit_test command to test_ * build: update e2e test and profile * docs: update development docs

* colab benchmark * add link * add link * update with colab badge * precommit * renamed folder name to notebooks * update link

* add rust linter * pre-commit Signed-off-by: Hsien-Cheng Huang <[email protected]> --------- Signed-off-by: Hsien-Cheng Huang <[email protected]>

* update link and position * update file position and cleanup script * colab link update

) * refactor: introduce 2 traits for flexible io type w/ example * linter * remove redundent test & update readme * remove numpy effort * remove numpy * remove numpy * fix test * no numpy * cleanup * fix imports * pre-commit and usize overflow check

Signed-off-by: 400Ping <[email protected]>

* feat: use rust hook for precommit * Refactor clippy hook in .pre-commit-config.yaml Updated clippy hook configuration in pre-commit. * run linter

* feat: [QDP] PyTorch Tensor Detection and CPU Path Signed-off-by: Cheyu Wu <[email protected]> * style: fix linter issue Signed-off-by: Cheyu Wu <[email protected]> * style: linter err Signed-off-by: Cheyu Wu <[email protected]> * doc: add comment for followup pr Signed-off-by: Cheyu Wu <[email protected]> * revert: ipynb linter fix Signed-off-by: Cheyu Wu <[email protected]> --------- Signed-off-by: Cheyu Wu <[email protected]>

…751) * Double-buffered async I/O for read_parquet_batch Signed-off-by: 400Ping <[email protected]> * update Signed-off-by: 400Ping <[email protected]> * fix python binding error Signed-off-by: 400Ping <[email protected]> * update Signed-off-by: 400Ping <[email protected]> * update Signed-off-by: 400Ping <[email protected]> * update Signed-off-by: 400Ping <[email protected]> * update Signed-off-by: 400Ping <[email protected]> * update Signed-off-by: 400Ping <[email protected]> * fix build error Signed-off-by: 400Ping <[email protected]> * Revert "fix build error" This reverts commit 3556b5a. * fix build errors Signed-off-by: 400Ping <[email protected]> * update unit test and boundary check Signed-off-by: 400Ping <[email protected]> * remove improvement 2 Signed-off-by: 400Ping <[email protected]> * fix qdp-core error Signed-off-by: 400Ping <[email protected]> * fix pre-commit Signed-off-by: 400Ping <[email protected]> * [Fix] fix pre-commit errors & warnings Signed-off-by: 400Ping <[email protected]> * fix rust linters Signed-off-by: 400Ping <[email protected]> * [Fix] handle buffer pool lock poisoning Signed-off-by: 400Ping <[email protected]> * [Chore] fix rust linters Signed-off-by: 400Ping <[email protected]> * update Signed-off-by: 400Ping <[email protected]> * Remove unused func Signed-off-by: 400Ping <[email protected]> * update Signed-off-by: 400Ping <[email protected]> --------- Signed-off-by: 400Ping <[email protected]>

* feat: add numpy as reader * add benchmark

guan404ming · 2026-01-04T17:53:34Z

After #778 and #779, let's do the final test for qdp and get this in!

ryankert01 · 2026-01-05T01:46:55Z

#790 fixed the pre-commit issue

guan404ming · 2026-01-05T02:15:14Z

Cherry-picked #790 to preserve the sequential branch and avoid rebasing it.

* The Scaling Test (Latency vs. Qubits) Signed-off-by: 400Ping <[email protected]> * fix pre-commit Signed-off-by: 400Ping <[email protected]> * [Chore] make initialization clearer & clearfy doc Signed-off-by: 400Ping <[email protected]> --------- Signed-off-by: 400Ping <[email protected]>

Signed-off-by: Hsien-Cheng Huang <[email protected]>

* [Core] Update throughput benchmark to batch encoding Signed-off-by: 400Ping <[email protected]> * fix conflict Signed-off-by: 400Ping <[email protected]> --------- Signed-off-by: 400Ping <[email protected]>

400Ping · 2026-01-06T06:40:18Z

website/community/index.md

 ## Biweekly Community Meeting
-Apache Mahout is committed to work consistently and conducts its meetings once in two weeks. To get invovled, you can access the link here. The meeting timings are communicated via official slack channel. 
+Apache Mahout is committed to work consistently and conducts its meetings once in two weeks. To get invovled, you can access the link here. The meeting timings are communicated via official slack channel.
 [Meeting Link](https://meet.google.com/hjo-njer-hzw?authuser=0&hs=122&ijlm=1767371505662&pli=1)


nit: Maybe changing this to subscribe to Mahout Google calendar.

Sure, let's do it in another pr

guan404ming · 2026-01-06T16:43:39Z

I’ve tested and verified all qdp features, and I believe this is ready to be merged into main. If any refinements are needed for QDP in the future, please open a PR targeting main.

400Ping · 2026-01-06T16:44:23Z

Nice

rich7420 and others added 30 commits November 28, 2025 13:09

[QDP] Initialize QDP, memory management, and DLPack protocol (#646)

9a96981

* structure * fix a lots of cuda errors * change folder name * improve

[QDP] Update pre-commit to add license headers for Rust (#660)

b44a4b5

[QDP] add unit test for qdp-core (#659)

c453f5f

[QDP] add integration test and introduction (#666)

7f78917

[QDP] improve memory initialization (#668)

957f569

[QDP] Add pre-processing module (#673)

038c9fd

* add preprocessing module * Remove unused import of Cow

[QDP] Add tests for pre-processing (#676)

ab2b6c9

Signed-off-by: 400Ping <[email protected]>

[QDP] make dataflow async and improve amplitudeEncoder (#675)

76db5a4

* async pipline * prevent error * Add TODO for dynamic chunk size tuning Added a TODO comment to tune chunk size dynamically based on GPU model or PCIe bandwidth. * update * add a test

[QDP] Update pre-commit to add license headers for Cuda (#684)

f7ce244

[QDP] add DataLoader Test (#687)

3cb194a

* Batch Throughput Signed-off-by: 400Ping <[email protected]> * fix Signed-off-by: 400Ping <[email protected]> * [chore] add comment Signed-off-by: 400Ping <[email protected]> --------- Signed-off-by: 400Ping <[email protected]>

[QDP] Gracefully handles OOM (#688)

5738b8d

* gracefully handles OOM Signed-off-by: 400Ping <[email protected]> * fix Signed-off-by: 400Ping <[email protected]> --------- Signed-off-by: 400Ping <[email protected]>

[QDP] Add fidelity test (#694)

a3aeeed

* add a fidelity test * fix format * reformat * effectively zero

[QDP] improve amplitudeEncoders for less copy memory allocations (#690)

a534f4c

* improve amplitudeEncoders for less copy memory allocations * Fix: apply pre-commit end-of-file-fixer * use pre-processing module

[QDP] update uv usage in qdp-python (#698)

7da8b90

Updated build instructions to include Python 3.11 environment setup.

[QDP] [Benchmark] Add a End-to-End benchmark to compare with speed (#697

335036a

) * [QDP] [Benchmark] Add a End-to-End benchmark to compare with speed * [QDP] improve memory management * Revert "[QDP] improve memory management" This reverts commit d909f3d.

[QDP] add batch kernel support (#700)

ce4b7ca

* [QDP] Add batch encoding support * Refactor batch pre-processing

[QDP] Add Arrow IPC support (#704)

2ff6664

* Add Arrow IPC support * Update default frameworks to "mahout-parquet" * Add tests for Arrow IPC * Add todo to prevent oom * Add TODO for supporting multiple float types

[QDP] Add data loader benchmark (#701)

a13a5c8

Co-Authored-By: Ping <[email protected]>

[QDP] GPU/Cuda kernel Optimizations (#706)

42397e0

* GPU/Cuda kernel Optimizations Signed-off-by: 400Ping <[email protected]> * fix Signed-off-by: 400Ping <[email protected]> --------- Signed-off-by: 400Ping <[email protected]>

[QDP] improve e2e benchmark (#713)

c614ee1

* improve e2e benchmark * gc.coolect first

[QDP] support turing architecture GPU (#731)

328eecc

[QDP] Choose between different frameworks in benchmark (#741)

97932d0

Signed-off-by: 400Ping <[email protected]>

[QDP] Enable 3060ti 3090ti cuda build sm86 (#737)

6992fc5

[docs][qdp] Add qdp development guide (#732)

f9e4c28

* docs: add development guide * refactor: precommit * docs: add dev container in DEVELOPMENT * docs: add more troubleshooting

rich7420 and others added 14 commits December 23, 2025 10:51

[QDP] Fix DLPack device_id hardcoding (#745)

aa30679

* [QDP] Fix DLPack device_id hardcoding# Please enter the commit message for your changes. Lines starting * update and move comments position * improve the test

[QDP] DLPack shape/strides: Support batch 2D tensor (#747)

03f71ec

[QDP] Add launch_amplitude_encode_f32 function (#739)

7a2b8fe

* float32 follow up Signed-off-by: 400Ping <[email protected]> * update Signed-off-by: 400Ping <[email protected]> --------- Signed-off-by: 400Ping <[email protected]>

[QDP] Add colab benchmark example (#755)

8f20ddf

* colab benchmark * add link * add link * update with colab badge * precommit * renamed folder name to notebooks * update link

[QDP] add Rust Linter to pre-commit hook (#758)

167f743

* add rust linter * pre-commit Signed-off-by: Hsien-Cheng Huang <[email protected]> --------- Signed-off-by: Hsien-Cheng Huang <[email protected]>

[QDP] Update notebooks dir and link (#762)

ef7e10d

* update link and position * update file position and cleanup script * colab link update

[QDP] Fix Route Error in Makefile (#775)

db71abf

[QDP] Add Workflow dispatch for manual triggering (#767)

febde8c

[QDP] Rename benchmark throughput script and refresh docs (#776)

dd3856b

Signed-off-by: 400Ping <[email protected]>

[QDP] Pre commit rust followup (#766)

3e781b8

* feat: use rust hook for precommit * Refactor clippy hook in .pre-commit-config.yaml Updated clippy hook configuration in pre-commit. * run linter

guan404ming changed the title ~~[QDP] Initiate QDP project~~ [QDP] Initiate project QDP Jan 4, 2026

[QDP] Numpy IO support (#777)

57f90e6

* feat: add numpy as reader * add benchmark

[QDP] fix pre-commit (#790)

c38412d

400Ping and others added 4 commits January 5, 2026 18:43

[QDP] [Numpy] add path exist check (#793)

7747bd3

Signed-off-by: Hsien-Cheng Huang <[email protected]>

[QDP] Fix benchmark to use batch encoding (#794)

4c99293

[QDP] Update benchmark_throughput to batch encoding (#796)

581f5ee

* [Core] Update throughput benchmark to batch encoding Signed-off-by: 400Ping <[email protected]> * fix conflict Signed-off-by: 400Ping <[email protected]> --------- Signed-off-by: 400Ping <[email protected]>

guan404ming marked this pull request as ready for review January 6, 2026 03:40

400Ping reviewed Jan 6, 2026

View reviewed changes

400Ping approved these changes Jan 6, 2026

View reviewed changes

guan404ming merged commit 5b6055a into main Jan 6, 2026
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[QDP] Initiate project QDP #786

[QDP] Initiate project QDP #786

guan404ming commented Jan 4, 2026

Uh oh!

guan404ming commented Jan 4, 2026

Uh oh!

ryankert01 commented Jan 5, 2026

Uh oh!

guan404ming commented Jan 5, 2026

Uh oh!

400Ping Jan 6, 2026

Uh oh!

guan404ming Jan 6, 2026

Uh oh!

guan404ming commented Jan 6, 2026 •

edited

Loading

Uh oh!

400Ping commented Jan 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[QDP] Initiate project QDP #786

[QDP] Initiate project QDP #786

Conversation

guan404ming commented Jan 4, 2026

Purpose of PR

Related Issues or PRs

Changes Made

Breaking Changes

Checklist

Uh oh!

guan404ming commented Jan 4, 2026

Uh oh!

ryankert01 commented Jan 5, 2026

Uh oh!

guan404ming commented Jan 5, 2026

Uh oh!

400Ping Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

guan404ming Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

guan404ming commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

400Ping commented Jan 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

guan404ming commented Jan 6, 2026 •

edited

Loading