(0.12.4 cherry-pick): aws: support copies >5GB #562
Conversation
| let multipart_copy_threshold = self | ||
| .multipart_copy_threshold | ||
| .map(|val| val.get()) | ||
| .transpose()? | ||
| .unwrap_or(MAX_SINGLE_REQUEST_COPY_SIZE); | ||
| let multipart_copy_part_size = self | ||
| .multipart_copy_part_size | ||
| .map(|val| val.get()) | ||
| .transpose()? | ||
| .unwrap_or(MAX_SINGLE_REQUEST_COPY_SIZE); |
There was a problem hiding this comment.
Clamp to 5GiB because that's the documented maximum?
There was a problem hiding this comment.
I think if someone wants to push it over 5GB, they should be able to. There are many "s3-compatible" object stores that might not share the same limitations.
| // Determine source size to decide between single CopyObject and multipart copy | ||
| let head_meta = self | ||
| .client | ||
| .get_opts( | ||
| from, | ||
| GetOptions { | ||
| head: true, | ||
| ..Default::default() | ||
| }, | ||
| ) | ||
| .await? | ||
| .meta; |
There was a problem hiding this comment.
Should this first try to use CopyObject and then fall back if it fails due to size?
There was a problem hiding this comment.
Good point - let me see if that's straightforward.
There was a problem hiding this comment.
The issue with that approach is that on error, AWS does not respond with anything more specific than "InvalidRequest":
<Error><Code>InvalidRequest</Code><Message>The specified copy source is larger than the maximum allowable size for a copy source: 5368709120</Message><RequestId>8550KAYYHRYF33SM</RequestId><HostId>R7zaiPWt96z/yQm2PtDT+pyFmYF76YCBcW0AeukdrXpS4qlSuO1nmXTFI4Ak2YcHMsBoymw33j4=</HostId></Error>
So there's not really a stable API for determining that the request is invalid because of the size of the source.
james-rms
force-pushed
the
jrms/0.12.4-aws-copy-etc
branch
from
5608942 to
68b24de
Compare
james-rms
force-pushed
the
jrms/0.12.4-aws-copy-etc
branch
from
6e4216a to
7fde896
Compare
2 participants
Which issue does this PR close?
Closes #.
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?