September 26, 2025: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The past 7 days saw 38 merged PRs and 29 open PRs across Velox, ClickHouse, Flink, and infrastructure areas. Major themes include Delta Lake native write support, Flink Nexmark benchmark expansion, daily Velox version synchronization, and performance optimizations for hash joins and window operations.

Key Ongoing Projects

• Delta Lake Native Write - @zhztheplayer leads the effort to add native Delta Lake write support for Spark 3.5 + Delta 3.3, with [GLUTEN-10215][VL] Delta: Native write support for Delta 3.3.1 / Spark 3.5 #10801 and [VL] Delta: Add Delta Lake write unit test for Spark 3.5 + Delta 3.3 #10802 providing the core implementation and test coverage
• Flink Nexmark Benchmark - @shuai-xu and @KevinyhZou continue expanding Nexmark query coverage, recently adding tests for q18-q21 ([GLUTEN-10550][FLINK] Add nexmark tests for q18-q21 #10757) and decimal type support ([GLUTEN-10628][FLINK] Add decimal type support to greaterthan function #10769)
• Daily Velox Sync - @GlutenPerfBot maintains daily version updates ([GLUTEN-6887][VL] Daily Update Velox Version (2025_09_26) #10808, [GLUTEN-6887][VL] Daily Update Velox Version (2025_09_25) #10800, [GLUTEN-6887][VL] Daily Update Velox Version (2025_09_24) #10792) keeping the Velox backend current with upstream improvements
• Performance Optimization - Multiple contributors focus on hash join performance ([VL] Setting higher value of Broadcast Hash Join threshold is degrading the performance of Spark with Gluten #7548), window operator streaming conversion ([GLUTEN-10731][VL]Remove the sort window in Velox backend #10734), and memory usage improvements ([CORE] Defer Protobuf serialization of SplitInfos in GlutenPartitions #10662)

Priority Items

• Release 1.5.0 Preparation - @PHILO-HE coordinates final patches for the upcoming release (Gluten 1.5.0 Release Preparation: Patch Porting, Documentation Updates, and Final Tasks #10574), including documentation updates ([DOC] Improve and simplify README.md and NewToGluten.md #10793)
• Critical Bug Fixes - @lgbo-ustc addresses ClickHouse crashes during task cancellation ([CH] Crashes caused by abnormal task cancellation. #10797, [GLUTEN-10772][CH] Fix crash caused by not releasing QueryContext resources properly. #10775), while @beliefer fixes performance regression by changing propagateIgnoreNullKeys default ([GLUTEN-10810][VL] Change the default value of propagateIgnoreNullKeys to false #10811)
• Build Issues - @jinchengchenghh works on Iceberg jar compatibility issues ([CH] Build failed by iceberg jar #10804) affecting ARM builds

Notable Discussions

• September 19, 2025: Weekly Status Update in Gluten #10759: Weekly status updates show strong momentum across all backends with 38 merged PRs
• [VL] Delta Lake write #10215: Delta Lake write support roadmap outlines remaining work for Spark 4.0 compatibility and native statistics tracking
• [VL] Setting higher value of Broadcast Hash Join threshold is degrading the performance of Spark with Gluten #7548: Broadcast hash join performance degradation investigation reveals optimization opportunities

Emerging Trends

• Lake-house Acceleration - Increased focus on native read/write support for Delta Lake and Iceberg formats
• Micro-performance Focus - Detailed optimizations in hash table builds, lazy vector loading metrics, and memory pool management
• Multi-backend Convergence - Efforts to unify features across Velox and ClickHouse backends
• Flink Integration Maturity - Comprehensive Nexmark benchmark support demonstrates production readiness

Good First Issues

• [CH] support expression MakeYMInterval #6814: Implement MakeYMInterval for ClickHouse - straightforward date interval function implementation
• [CH] support function date_from_unix_date  #4730: Add date_from_unix_date function - follows existing date function patterns in ClickHouse backend
• [CH] support split_part function #6807: Support split_part string function - basic string manipulation suitable for learning CH UDF framework
• [CH] support function SparkPartitionID  #6812: Implement SparkPartitionID function - reuses existing Spark partition ID functionality
• [CH] support expression MapZipWith #6815: Add MapZipWith expression support - entry-level map function implementation

These issues require basic C++ knowledge and ClickHouse function registration understanding, making them ideal for newcomers to contribute to the project.