feat: slicing strategy for blob store operations

[wbbradley]

Description

Enable automatic file slicing during walrus store.

Test plan

Manual testing, CI Pipeline.

---

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that
a user might notice and any actions they must take to implement updates. (Add release notes after the colon for each item)

• Storage node:
• Aggregator:
• Publisher:
• CLI: Added --slice-size argument to the walrus store command that allows for slicing very large files into smaller chunks. The workflow here is to utilize this, then upon fetching all of the stored blobs (in order), concatenating them back together. This is a bit of an advanced procedure for folks with more complex requirements. Note that this is a building block that future features can take advantage of.

[github-actions]

Warning: This PR modifies the Walrus CLI. Please consider the following:

• Make sure the changes are backwards compatible. Consider deprecating options before
  removing them.
• Generally only use long CLI options, not short ones to avoid conflicts in the
  future.
• If you added new options or features, or modified the behavior, please document the
  changes in the release notes of the PR and update the documentation in the docs/book
  directory.

[jarrvis]

Hi 👋
Nice to see this feature coming in!

I want to ask about two things:

• just to confirm: is the client responsible for remembering the blobIds that compose the sliced blob (and the order of those)?

• do you see a possibility of uploading the slices as quilt patches? Possibly in multiple quilts? That could "unlock" using tiny slice sizes, which may allow for smooth streaming when aggregating

[wbbradley]

Hey Michal!

• just to confirm: is the client responsible for remembering the blobIds

Yes, at this time the design is geared towards simplicity in that manner. It's not crazy to think that this could be extended to include an indrection through another "index" or "manifest" blob, or simply an object on-chain, although those do have various tradeoffs to consider.

• do you see a possibility of uploading the slices as quilt patches?

I'm also working on a concatenated streaming api in the aggregator. The way I'm looking at it, the design of these large-blob features really depends on how they will eventually be consumed. I think as a next step after the basic blob-level granularity it makes sense to try quilt patches as chunks. That could be a good idea, but it will take some careful measurement to understand the trade-offs between storage efficiency and latency to first-read, etc.

[github-actions]

This PR is stale because it has been open 14 days with no activity. It will be closed in 7 days unless you remove the stale label, add the do-not-close label, or comment on it.

[wbbradley]

Keep alive for now.

[github-actions]

This PR is stale because it has been open 14 days with no activity. It will be closed in 7 days unless you remove the stale label, add the do-not-close label, or comment on it.