[Feature Request] Split Large ‘tar’ Archive into Multiple Files of Certain Size

[thangckt]

Dear developers,

Can you give an option to split a large .tar file into smaller parts before downloading?

It will more efficient for transferring.

I face a problem that the backward files have large size, then it can not be downloaded, even dpdispatcher finishs without error.

Thank

[felix5572]

for temp use just modify dpdispatcher/contexts/ssh_context.py _get_file method? (tar_command and split command).

For example

# Step 1: Create a tar archive
tar -cvf archive.tar largefile.txt

# Step 2: Split the tar archive into 100 MB chunks
split -b 100M archive.tar archive_part_

# Step 3: Combine the split files back into a single tar file
cat archive_part_* > archive.tar

# Step 4: Extract the contents of the tar file
tar -xvf archive.tar