Backend-specific contents in common code spaces

[zhztheplayer]

@apache/gluten-committers

We now have some code that is for VL / CH backend specifically, but placed in common code module. A typical one we are all aware of is backends' CPP code. Velox's is under cpp while CH's is under cpp-ch. I am looking for a good timing to move them to their own modules (backends-velox / backends-clickhouse).

If you have some thoughts on this, please comment or reply to this email thread. Thanks.

[PHILO-HE]

@zhztheplayer, as backends-velox / backends-clickhouse are maven modules, not sure whether it is proper to move cpp code into them, although it should be technically feasible to do that.

[zhztheplayer]

@zhztheplayer, as backends-velox / backends-clickhouse are maven modules, not sure whether it is proper to move cpp code into them, although it should be technically feasible to do that.

Thank you for the insights.

Our CPP code tightly relies on JNI so can't be individually workable without the upper layer arch or Maven/Java code. So the code, for example cpp/ is in a way something that belongs to Velox backend's Java module from my perspective.

If there is a chance we can split out the common code from cpp/, then it's feasible to have a common place for it.

cpp/ / cpp-ch/ are big deals but can be easy to move. We can make the decision later anyway.

The non-trivial one is UT module. I think there is no easy way to move them to backend modules without large scale refactors.

[zhztheplayer]

cc @zzcclp @baibaichen If you have any thoughts.

[baibaichen]

I agree to reorganize the code, but can we do this refactoring after the release of 1.3?

What's the problem of UT module?

[zhztheplayer]

I agree to reorganize the code, but can we do this refactoring after the release of 1.3?

Thanks! I don't have enough time for doing this immediately. If no one takes the work on, it will not likely be in 1.3. Let's just keep an eye on the contents and avoid adding more.

What's the problem of UT module?

gluten-ut relies on backends-velox / backends-clickhouse and has some if-else conditions on backend type written in the module. Ideally, refactors needed to move some of the non-common logics to their own modules.

[baibaichen]

Yes. I also find this issue for gluten-ut. we can start to sovle this.

[FelixYBW]

I'd think to keep the gluten/cpp folder, to isolate the java/scala folder. But since we have nothing cpp code shared between VL and CH, it's more clear to rename the gluten/cpp to gluten/cpp-vl. In future if we do be able to share some cpp code, we can merge both.

[zhztheplayer]

it's more clear to rename the gluten/cpp to gluten/cpp-vl. In future if we do be able to share some cpp code, we can merge both.

CH and VL will not likely share CPP code in short term. Velox's CPP code, namely gluten/cpp, has an extensive design but I have been keeping questing it on its effectivity for our architecture so far. Technically some of the code can be placed into gluten-arrow as it was designed for common purpose but it's not that optimal putting CPP code in root layer anyway, speaking of our current code structure.

I will not exclude the probability that we start to incubate an individual CPP project for some reason in future, but that could be a different story. So it's safe for us to focus on Gluten project itself at the moment.

[FelixYBW]

@zhztheplayer can you create some google doc to more concrete plan? We may setup a meeting with @baibaichen to go through it.

[Yohahaha]

I vote for move cpp module to backend-velox module, then move ep/build-velox under cpp, and we can use git submodule to build link with Velox.

[FelixYBW]

The final goal is to share some cpp code as well.

[FelixYBW]

@zhztheplayer Can you fill the table? Does all the gluten- folder hold CK and VL code? in the code do we have lots of if CK xxx elif VL yyy? We should avoid to use the if else in code. I remember we have cleaned once before. Instead to put them in different folder, like ep/build-clickhouse and ep/build-velox

	CK code	VL code	Java	CPP	comments
backends-clickhouse	Y		Y
backends-velox		Y	Y
cpp-ch	Y			Y
cpp		Y		Y
dev
docs
ep	Y	Y		Y
gluten-arrow
gluten-celeborn
gluten-core
gluten-delta		Y
gluten-hudi		Y
gluten-iceberg		Y
gluten-ras
gluten-substrait
gluten-ui
gluten-uniffle
gluten-ut	Y	Y
package
shims
tools

[FelixYBW]

Let's make clear what we shared today and what we can share in future. The final goal is to offload to any libraries like a gpu backend.

[zhztheplayer]

The top-level data lake support modules are all for Velox backend IIRC. They should be trivial to move and we can get started from this part.