Snappy for compression codec using velox as backend #6697
Unanswered
himanshu-zetta
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I need to run gluten on spark with snappy as shuffle compression codec.
As gluten velox backends api support only lz4 and zstd as shown in the figure below:
so, I added snappy in list using this Set("lz4", "zstd", "snappy").
This compiled and I was able to run the spark queries with spark.io.compression.codec=snappy.
But the issue I'm facing which seems like there is no compression happening during snappy with gluten as when I compared the shuffle write with base I observed ~2x write in case of gluten.
@PHILO-HE , @FelixYBW, @weiting-chen could you help me, what else I need to do to use snappy codec for shuffle compression
Beta Was this translation helpful? Give feedback.
All reactions