September 19, 2025: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The past 7 days delivered 38 merged PRs and 29 open PRs. Velox backend dominates with daily version bumps, GPU/cuDF integration, and window-operator refactoring. Flink activity centers on Nexmark benchmark coverage and UDF support. Iceberg/Delta lake features continue to mature, while build hygiene and CI improvements keep pace.

Key Ongoing Projects

• Daily Velox sync – @GlutenPerfBot lands fresh commits every day ([GLUTEN-6887][VL] Daily Update Velox Version (2025_09_19) #10758, [GLUTEN-6887][VL] Daily Update Velox Version (2025_09_18) #10749, [GLUTEN-6887][VL] Daily Update Velox Version (2025_09_17) #10730, [GLUTEN-6887][VL] Daily Update Velox Version (2025_09_16) #10720, [GLUTEN-6887][VL] Daily Update Velox Version (2025_09_15) #10711) keeping the Velox backend on the bleeding edge.
• GPU/cuDF connector – @jinchengchenghh adds single-GPU task locking ([GLUTEN-10683][VL] Add lock to restrict single task for one GPU #10684) and cuDF parquet sink ([GLUTEN-8072][VL] Add close method to close the hdfs connection #10593); validation logic still being refined (Optimize cudf validation logic #10753).
• Window-operator refactor – @JkSelf removes SortWindow in favor of streaming-only execution ([GLUTEN-10731][VL]Remove the sort window in Velox backend #10734, Remove the SortWindow type in Velox backend. #10731) with follow-up performance tweaks ([GLUTEN-10666][VL]Optimize sort window to streaming window if child order matchs #10667).
• Delta Lake PoC – @zhztheplayer prototypes native write ([GLUTEN-10215][VL] PoC: Delta Lake write for Spark 3.5.2 / Delta 3.3.1 #10216) and deletion-vector read ([VL] Delta deletion vector read support for Spark 3.4 + Delta 2.4 #10740) for Delta 2.4/3.3.
• Flink Nexmark sprint – @shuai-xu & @KevinyhZou add q11-q22 coverage, UDFs (count_char, date_format) and decimal support ([GLUTEN-10632][FLINK] Add nexmark test for q11, q12 and q22 #10735, [GLUTEN-10550][FLINK] Add nexmark tests for q18-q21 #10757, [GLUTEN-10247][FLINK] Support date_format function #10248, [FLINK] greater than support decimal type #10628).

Priority Items

• Release 1.5.0 blockers – @PHILO-HE tracking final back-ports (Gluten 1.5.0 Release Preparation: Patch Porting, Documentation Updates, and Final Tasks #10574); weekly build fixed by limiting Spark to 3.5 ([VL] Fix weekly build validation #10750).
• Critical correctness fixes – @lgbo-ustc fixes empty aggregation keys in CH GroupLimit ([GLUTEN-10741][CH] Invalid results: sort keys contains partition keys when get windows' topK #10746); @Zouxxyy eliminates redundant c2r/r2c for Iceberg partition write ([GLUTEN-10713] Eliminate the c2r before iceberg partition write #10714).
• Memory leak & OOM – [VL] Memory Leak possibly TableScan #9456 TableScan leak under active investigation; After hash join, the size of the generated table is three times that of vanilla Spark #10693 3× table-size blow-up after hash-join needs triage.
• Flaky tests – @jinchengchenghh stabilizes CH adaptive-query suite ([CH] Flaky test ClickHouseAdaptiveQueryExecSuite #10756).

Notable Discussions

• Add a new backend: Omni #10188: @wjunLu proposes new ARM-optimized Omni backend—community feedback invited on GPIP doc.
• [Flink] question about RowKind in results #10717: @ryyyyyy1 asks how Flink’s RowKind (+I/+U/-U) should map to Velox RowVector—design open.
• Gluten Slack Channel #incubator-gluten #8429: Slack channel #incubator-gluten now live—ASF members and guests welcome.

Emerging Trends

• Lake-house acceleration – daily PRs for Iceberg/Delta read/write, deletion vectors, column mapping.
• Micro-performance focus – hash-table build configs ([GLUTEN-10660][VL] Adding configuration for hash table build #10634), lazy-vector metrics ([VL] Add metrics for lazy vector load #10726), batch-size soft limits ([VL] Support Velox's preferred_output_batch_bytes config #10661).
• Build hygiene – spotless POM enforcement ([MINOR] Enable pom spotless format check #10755), Scala format checker ([VL][CI] Add a format checker for scala code to enable early failure reporting #10747), Spark 4.0 CI readiness ([GLUTEN-10207][VL] adding spark40 unit tests #10725).

Good First Issues

• [CH] support expression MakeYMInterval #6814: implement MakeYMInterval for ClickHouse—pure CH backend, no native code.
• [CH] support function date_from_unix_date  #4730: add date_from_unix_date CH function—follow existing date-function pattern.
• [CH] support split_part function #6807: support split_part string function in CH—straightforward string splitting.
• [CH] support function SparkPartitionID  #6812: expose SparkPartitionID in CH backend—reuse Spark’s partition ID.
• [CH] support expression MapZipWith #6815: implement MapZipWith for CH—entry-level map function, great for learning CH UDF framework.

All CH good-first issues need basic C++ and ClickHouse function registration; unit tests & docs expected.