Distributed Compaction support in Engine (Ray, Spark)

[yingjianwu98]

Compaction currently only happens in a single node with multiple threads. Adding distributed compaction capabilities to Lance's distributed execution engines enables users to leverage multiple machines in a cluster to parallelize compaction.

If the community feels this feature is useful, I will come up with a detailed working plan on this, thanks!

I have created a ticket as well. #5040 (comment)