October 31, 2025: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The past 7 days saw 32 merged PRs and 20 open PRs, with heavy focus on Velox backend stability, GPU/cuDF acceleration, and daily upstream Velox syncs. Memory-management fixes and shuffle optimizations dominated the bug-fix queue, while new backend proposals (Bolt, Omni) sparked community interest.

Key Ongoing Projects

• GPU shuffle & cuDF acceleration – @jinchengchenghh is leading [GLUTEN-10933][VL] cuDF: Move lock to ShuffleReader #10934 and [VL] Support GPU shuffle reader #10933 to move locks out of the iterator constructor, enabling 1 GB GPU batches and concurrent CPU/GPU pipeline preparation.
• Daily Velox up-streaming – @GlutenPerfBot continues the mechanical daily version bumps ([GLUTEN-6887][VL] Daily Update Velox Version (2025_10_31) #10987, [GLUTEN-6887][VL] Daily Update Velox Version (2025_10_30) #10985, [GLUTEN-6887][VL] Daily Update Velox Version (2025_10_30) #10978, [GLUTEN-6887][VL] Daily Update Velox Version (2025_10_29) #10974, [GLUTEN-6887][VL] Daily Update Velox Version (2025_10_28) #10962, [GLUTEN-6887][VL] Daily Update Velox Version (2025_10_26) #10949, [GLUTEN-6887][VL] Daily Update Velox Version (2025_10_25) #10947, [GLUTEN-6887][VL] Daily Update Velox Version (2025_10_25) #10946) keeping Gluten in lock-step with facebookincubator/velox.
• Apache maturity checklist – @zhztheplayer closed [CORE] Implement stage-level resourceProfile auto-adjust framework to avoid oom #8018 (release-process docs) and [VL] Delta Lake read enhancements #10377, pushing the podling toward graduation.
• BHJ hash-table broadcast – @JkSelf’s 1.6 k-line PR [GLUTEN-7548][VL] Optimize BHJ in velox backend #8931 (open, 182 comments) promises 1.29× TPC-DS Q23a speed-up and OOM relief for Q24a/b.
• ClickHouse backend refresh – @lgbo-ustc posted [GLUTEN-10727][CH]Daily Update Clickhouse Version (20250904) #10728, the monthly ClickHouse version update.

Priority Items

• Memory regression – [VL] Unexpected behavior DynamicOffHeapSizingMemoryTarget.ASYNC_GC_SUSPEND #10937 (open) reports spill can’t be triggered when dynamic off-heap sizing is on; @wForget already has a fix in review ([GLUTEN-10932][VL] Pass memory changes to taskMemoryManager for dynamic off heap #10936).
• ORC schema mismatch – long-standing [VL] Read OrcFile error when schema in Orc file and the table file don't consist #5638 (open) breaks reads when ORC file lacks column names; @ccat3z’s PR [VL] Support read old ORC file without column names #8862 is stalled awaiting upstream Velox reviews.
• Uniffle shuffle performance – [VL] Performance regression on uniffle hash shuffle reader #10920 (open) flags a 1.5.0 regression vs 1.2.0; @wForget’s buffer-size knob [GLUTEN-10920][VL] Allow disabling hash/sort shuffle reader buffer #10922 was merged but more tuning is expected.
• GCC-13 readiness – [VL] Prepare for GCC-13 upgrade #10926 (open) reminds us Velox will soon require GCC-13; CI still on CentOS-7 + GCC-11.

Notable Discussions

• Add a new backend: Bolt #10929: @WangGuangxin proposes “Bolt”, a ByteDance Velox fork with JIT and OOM hardening, asking for guidance on upstreaming it as a new Gluten backend.
• Add a new backend: Omni #10188: @wjunLu presents “Omni”, an ARM-optimized backend showing 70 % TPC-DS speed-up; the team offers ARM CI resources if merged.

Emerging Trends

• GPU-first features – cuDF validation, GPU shuffle reader, and cudf library pre-installs are landing almost daily, signaling a shift from CPU-only Velox.
• Memory-management churn – three separate dynamic off-heap fixes in one week suggest the feature is newly stressed in production.
• Multi-backend ecosystem – with Bolt and Omni proposals, Gluten is evolving into a thin meta-layer over pluggable native engines.
• Graduation push – documentation PRs for release process, security pages, and PMC lists indicate serious TLP submission prep for Q4 2025.

Good First Issues

• [CH] support expression MakeYMInterval #6814: Add ClickHouse expression MakeYMInterval – pure CH backend, no Velox/C++ needed.
• [CH] support function date_from_unix_date  #4730: Implement date_from_unix_date for ClickHouse – similar pattern to existing date functions.
• [CH] support split_part function #6807: Add split_part string function for ClickHouse – well-scoped, CH-only.
• [CH] support function SparkPartitionID  #6812: Implement SparkPartitionID for ClickHouse – single-row function, good intro to CH function registry.
• [CH] support expression MapZipWith #6815: Add MapZipWith higher-order function for ClickHouse – slightly larger, but excellent for learning CH’s lambda infrastructure.

All issues above are labeled “good first issue”, touch only the ClickHouse backend, and have clear function signatures, making them ideal entry points for new contributors comfortable with Java/Scala or C++ in ClickHouse context.