October 03, 2025: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The past 7 days have been dominated by release-prep fever: 1.5.0 is being finalized, so version-bump PRs, doc polish and back-ports are everywhere. Meanwhile Velox keeps moving (daily bumps), Iceberg & Delta lake-house features are expanding, and a long tail of micro-optimizations (memcpy, vector copy, hash-table configs) show the community is squeezing every last cycle out of the engine.

Key Ongoing Projects

• 1.5.0 release – @PHILO-HE drove [GLUTEN-10574][1.5] Bump version to 1.5.0 to finalize release #10829 (version bump) and [GLUTEN-10574][1.5] Backport #10793 and #10807 to update documents and automate release process #10827 (release-automation back-ports); @zhouyuan added Spark-4.0 unit-test suite ([GLUTEN-10207][VL] adding spark40 unit tests #10725, 56 k lines!).
• Native Delta/Iceberg write – @zhztheplayer’s 1 787-line PoC ([GLUTEN-10215][VL] Delta: Native write support for Delta 3.3.1 / Spark 3.5 #10801) plus companion fixes ([VL] Delta: Fix input file expressions not getting pushed to native scan when column mapping is applicable #10822, [VL] Following #10823, correct the config option key #10830) bring off-loaded Parquet writer for Delta 3.3 on Spark 3.5; @Zouxxyy landed dynamic-partition overwrite for Iceberg ([GLUTEN-10648][VL] Support Iceberg overwrite partitions dynamic #10823, [GLUTEN-10648][VL] Support Iceberg overwrite partitions dynamic #10760).
• Velox daily sync – @GlutenPerfBot keeps us current with upstream ([GLUTEN-6887][VL] Daily Update Velox Version (2025_10_02) #10832, [GLUTEN-6887][VL] Daily Update Velox Version (2025_10_03) #10837, [GLUTEN-6887][VL] Daily Update Velox Version (2025_10_01) #10826, [GLUTEN-6887][VL] Daily Update Velox Version (2025_09_27) #10814).
• Omni-backend proposal – @wjunLu opened Add a new backend: Omni #10188 to add an ARM-optimized OmniOperator backend (70 % TPC-DS speed-up claimed).

Priority Items

• Performance regression – [GLUTEN-10810][VL] Change the default value of propagateIgnoreNullKeys to false #10811 by @beliefer flips propagateIgnoreNullKeys default after TPC-DS q87-90 slowdown; needs review.
• Column-to-row memcpy – [GLUTEN-10833][VL] Reduce memcpy in ColumnToRow #10824 by @zhouyuan and [VL] Reduce vector copy when doing c2r #10825 by @zhli1142015 both attack redundant copies; quick wins, low risk.
• Build stability – [CH] Build failed by iceberg jar #10804 (Iceberg jar version clash on ARM) closed but similar flakes in [CH] Flaky test ClickHouseAdaptiveQueryExecSuite #10756 (CH AQE test) still open; @jinchengchenghh and @lgbo-ustc are chasing.
• Input-file expressions – Support input file expressions for all table formats #10840 tracks missing Delta/hudi support after [CORE] Iceberg: Implement input file expressions for Iceberg tables #10831 fixed Iceberg; @JunhyungSong has PR [CORE] Iceberg: Implement input file expressions for Iceberg tables #10831 open and needs follow-ups.

Notable Discussions

• Add a new backend: Omni #10188: “Add a new backend: Omni” – ARM-centric accelerator; community weighing maintenance & CI resourcing.
• September 26, 2025: Weekly Status Update in Gluten #10813: weekly status bot summary (38 merged PRs, 29 open) – shows sustained velocity across Velox/CH/Flink.

Emerging Trends

1. Lake-house parity race – Iceberg & Delta getting simultaneous native read/write features; expect similar push for Hudi.
2. Micro-perf everywhere – C2R memory zeroing, vector flatten avoidance, hash-table tuning: death-by-a-thousand-cuts phase.
3. Multi-backend convergence – same feature (input-file expr, cudf connector, config API) implemented for both Velox and CH within days.
4. Release automation – new GHA workflows ([GLUTEN-9615][INFRA] Automate release process with build script and GHA workflow #10807, [GLUTEN-10574][1.5] Backport #10793 and #10807 to update documents and automate release process #10827) and CentOS-7 branch checkout ([CI] Check out PR branch in the weekly CentOS 7 build workflow to ease PR verification #10839) show infra maturing.

Good First Issues

• [CH] support expression MakeYMInterval #6814: implement MakeYMInterval for ClickHouse – simple date-interval UDF, great to learn CH function registry.
• [CH] support function date_from_unix_date  #4730: add date_from_unix_date – follows existing date-function pattern.
• [CH] support split_part function #6807: support split_part string function – self-contained, touches parser + UDF stub.
• [CH] support function SparkPartitionID  #6812: expose SparkPartitionID – reuses existing partition metadata, no native code needed.
• [CH] support expression MapZipWith #6815: add MapZipWith – entry-level map function; good excuse to peek into Velox/CH map kernels.

All need basic C++ or Scala, come with clear signatures, and have prior examples to copy.