October 10, 2025: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The past 7 days have been dominated by release-prep fever: 1.5.0 is being finalized, so version-bump PRs, doc polish and back-ports are everywhere. Meanwhile Velox keeps moving (daily bumps), Iceberg & Delta lake-house features are expanding, and a long tail of micro-optimizations (memcpy, vector copy, hash-table configs) show the community is squeezing every last cycle out of the engine.

Key Ongoing Projects

• 1.5.0 release – @PHILO-HE drove [GLUTEN-10574] Include copyright for Hudi and Paimon in NOTICE-binary #10855 (copyright NOTICE) and [GLUTEN-10574] Correct Gluten versions in documents and jupyter notebooks #10858 (doc version fixes); @zhztheplayer updated Flink docs for RC1 ([1.5][FLINK] Update Jar versions in Flink.md to match release 1.5 #10861)
• Native Delta/Iceberg write – @zhztheplayer’s 1 787-line PoC ([GLUTEN-10215][VL] Delta: Native write support for Delta 3.3.1 / Spark 3.5 #10801) plus companion fixes ([VL] Delta: Fix input file expressions not getting pushed to native scan when column mapping is applicable #10822, [VL] Following #10823, correct the config option key #10830) bring off-loaded Parquet writer for Delta 3.3 on Spark 3.5; @Zouxxyy landed dynamic-partition overwrite for Iceberg ([GLUTEN-10648][VL] Support Iceberg overwrite partitions dynamic #10823, [GLUTEN-10648][VL] Support Iceberg overwrite partitions dynamic #10760)
• Velox daily sync – @GlutenPerfBot keeps us current with upstream ([GLUTEN-6887][VL] Daily Update Velox Version (2025_10_10) #10862, [GLUTEN-6887][VL] Daily Update Velox Version (2025_10_09) #10854, [GLUTEN-6887][VL] Daily Update Velox Version (2025_10_08) #10849, [GLUTEN-6887][VL] Daily Update Velox Version (2025_10_07) #10847, [GLUTEN-6887][VL] Daily Update Velox Version (2025_10_06) #10844)
• Performance micro-optimizations – @zhouyuan and @zhli1142015 attack redundant memcpy in Column-to-Row ([GLUTEN-10833][VL] Reduce memcpy in ColumnToRow #10824, [VL] Reduce vector copy when doing c2r #10825); @zml1206 adds months_between function support ([GLUTEN-10781][VL] Add months_between support #10782)

Priority Items

• Performance regression – [GLUTEN-10810][VL] Change the default value of propagateIgnoreNullKeys to false #10811 by @beliefer flips propagateIgnoreNullKeys default after TPC-DS q87-90 slowdown; needs review
• Spill level bug – Spill Level Exceeds Maximum in HashBuild Operator #10845 reports HashBuild operator spill level exceeding maximum; @himanshu-goyal-hxh needs help
• OOM in UnboundedWindowFunctionFrame – [VL] UnboundedWindowFunctionFrame OOM when partition is large #10500 by @zml1206 affects entire-partition window queries; needs memory management fix
• Build stability – [CH] Build failed by iceberg jar #10804 (Iceberg jar version clash on ARM) closed but similar flakes in [CH] Flaky test ClickHouseAdaptiveQueryExecSuite #10756 (CH AQE test) still open; @jinchengchenghh and @lgbo-ustc are chasing

Notable Discussions

• October 03, 2025: Weekly Status Update in Gluten #10841: Weekly status bot summary shows sustained velocity across Velox/CH/Flink with 38 merged PRs, 29 open
• Stage level resource management to handle offheap/onheap memory conflict #4392: Stage level resource management discussion for offheap/onheap memory conflict handling
• [VL] upstream OAP/Velox commits to upstream #8782: OAP/Velox upstreaming tracking issue with multiple blocked PRs

Emerging Trends

• Lake-house parity race – Iceberg & Delta getting simultaneous native read/write features; expect similar push for Hudi
• Micro-perf everywhere – C2R memory zeroing, vector flatten avoidance, hash-table tuning: death-by-a-thousand-cuts phase
• Multi-backend convergence – same feature (input-file expr, cudf connector, config API) implemented for both Velox and CH within days
• Release automation – new GHA workflows and packaging fixes show infra maturing

Good First Issues

• [CH] support expression MakeYMInterval #6814: Implement MakeYMInterval for ClickHouse – simple date-interval UDF, great to learn CH function registry
• [CH] support function date_from_unix_date  #4730: Add date_from_unix_date – follows existing date-function pattern
• [CH] support split_part function #6807: Support split_part string function – self-contained, touches parser + UDF stub
• [CH] support function SparkPartitionID  #6812: Expose SparkPartitionID – reuses existing partition metadata, no native code needed
• [CH] support expression MapZipWith #6815: Add MapZipWith – entry-level map function; good excuse to peek into Velox/CH map kernels

All need basic C++ or Scala, come with clear signatures, and have prior examples to copy.