ldbc模块的相关问题

[NineSue]

@SemyonSinchenko 老师您好，我现在在做 GraphAr 集成 LDBC 的任务，目前已经实现了一部分 RDD 的处理和转换并能在内存中跑通，但遇到两个问题想请教您：

1.实现过程中必然需要调用 LDBC 的接口，但我在中央 Maven 仓库和阿里云 Maven 仓库都没有找到 ldbc_datagen_spark 的 jar 包。现在我是用比较笨的办法，把 LDBC 打成一个 jar 丢到本地 Maven，再导入到项目里，同时引入相应依赖。请问有没有更好、更优雅的办法？

2.关于模块放置的位置，我目前是放在 commons 模块下新建一个包，然后把跑的类放在 example 下。请问这样组织是否合适，还是应该单独建一个模块会更规范？

[NineSue]

@SemyonSinchenko
Hello Professor, I’m currently working on the GraphAr integration with LDBC task. I’ve already implemented part of the RDD processing and transformation, and it can run successfully in memory. However, I’ve encountered two issues I’d like to ask you about:

1.In the implementation process, it’s inevitable that we need to call the LDBC interface, but I couldn’t find the ldbc_datagen_spark jar package in either the Central Maven Repository or the Aliyun Maven Repository. For now, I’m using a rather naive approach: packaging LDBC into a jar, installing it into my local Maven, and then importing it into the project along with the necessary dependencies. Is there a better or more elegant solution?

2.Regarding module placement, I currently put it under the commons module by creating a new package, and I placed the runnable class under example. Do you think this organization is appropriate, or would it be more standard to create a separate module?

[SemyonSinchenko]

Hi! @NineSue
Could you clarify please, what exact part of LDBC do you need? As I remember, there are Graphalytics Benchmarks, Social Network Benchmark, Semantic Publishing Benchmark and maybe something else...

[NineSue]

Hi @SemyonSinchenko, thanks for your reply!

To clarify: I’m working specifically with the LDBC Social Network Benchmark (SNB) Datagen Spark module.
My goal is to enhance Apache GraphAr (incubating) so that it can directly consume the RDDs/DataFrames generated inside SNB Datagen Spark and export them in GraphAr format, instead of relying on intermediate CSV files.

At the moment, I’ve already implemented part of the RDD transformation logic and can run it in memory, but I’m facing two questions:

1.I couldn’t find an official ldbc_datagen_spark jar in Maven Central or Aliyun Maven. Currently, I’m doing it in a “manual” way: packaging the LDBC repo as a jar and installing it into my local Maven repo to import it. Do you know if there’s a more standard or elegant way to include this dependency?

2.For module placement: right now I put the code under the commons module by creating a new package, and placed runnable classes under example. Do you think this structure is appropriate, or would it be better to create a separate module dedicated to the LDBC integration?

Thanks a lot for your guidance!

[SemyonSinchenko]

For the p.1 it looks good for me. We can add it as a git submodule.

For the p.2: I think we should have a separate module to avoid a mess when we will be ready to publish Gar Spark to Maven Central. Otherwise it may be problematic to separate what should be published and what should not.

[NineSue]

@SemyonSinchenko, I see — understood, thank you very much for your reply! 🙏

[NineSue]

One more question — the separate module should be at the same level as commons, that’s fine, right?
@SemyonSinchenko

[SemyonSinchenko]

I think yes. It will simplify making Gar a runtime dependency of it when needed.

[NineSue]

I think yes. It will simplify making Gar a runtime dependency of it when needed.

ok,thanks