[s3] iter_bucket fails with 403 unauthorised when using moto #637
Description
Problem description
I'm attempting to implement tests for an application using smart_open.s3.iter_bucket to read files from an S3 bucket. In the tests I am using moto to mock S3 (in a similar way to the smart_open tests themselves). However when iter_bucket calls bucket.download_fileobj(key_name, buf) it is returning 403 unauthorised. I am able to make this call myself from within the tests without issue.
Steps/code to reproduce the problem
Example test that produces this issue on my system (based on smart_open tests):
@moto.mock_s3
class IterBucketTest(unittest.TestCase):
def setUp(self):
ignore_resource_warnings()
def tearDown(self):
cleanup_bucket()
def test_iter_bucket(self):
populate_bucket()
results = list(smart_open.s3.iter_bucket(BUCKET_NAME))
self.assertEqual(len(results), 10)Result of running test_iter_bucket:
test_iter_bucket (common_test.smart_open_test.IterBucketTest) ... ERROR
NoneType: None
======================================================================
ERROR: test_iter_bucket (common_test.smart_open_test.IterBucketTest)
----------------------------------------------------------------------
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/project/path/venv/lib/python3.9/site-packages/smart_open/s3.py", line 1254, in _download_key
content_bytes = _download_fileobj(bucket, key_name)
File "/project/path/venv/lib/python3.9/site-packages/smart_open/s3.py", line 1271, in _download_fileobj
bucket.download_fileobj(key_name, buf)
File "/project/path/venv/lib/python3.9/site-packages/boto3/s3/inject.py", line 718, in bucket_download_fileobj
return self.meta.client.download_fileobj(
File "/project/path/venv/lib/python3.9/site-packages/boto3/s3/inject.py", line 678, in download_fileobj
return future.result()
File "/project/path/venv/lib/python3.9/site-packages/s3transfer/futures.py", line 106, in result
return self._coordinator.result()
File "/project/path/venv/lib/python3.9/site-packages/s3transfer/futures.py", line 265, in result
raise self._exception
File "/project/path/venv/lib/python3.9/site-packages/s3transfer/tasks.py", line 255, in _main
self._submit(transfer_future=transfer_future, **kwargs)
File "/project/path/venv/lib/python3.9/site-packages/s3transfer/download.py", line 340, in _submit
response = client.head_object(
File "/project/path/venv/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/project/path/venv/lib/python3.9/site-packages/botocore/client.py", line 705, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/project/path/venv/lib/python3.9/site-packages/moto/core/models.py", line 111, in wrapper
result = func(*args, **kwargs)
File "/project/path/test/common_test/smart_open_test.py", line 49, in test_iter_bucket
results = list(smart_open.s3.iter_bucket(BUCKET_NAME))
File "/project/path/venv/lib/python3.9/site-packages/smart_open/s3.py", line 1192, in iter_bucket
for key_no, (key, content) in enumerate(result_iterator):
File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 870, in next
raise value
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
----------------------------------------------------------------------
Ran 1 test in 4.255s
FAILED (errors=1)
Versions
macOS-11.4-x86_64-i386-64bit
Python 3.9.2 (default, Feb 24 2021, 13:26:09)
[Clang 12.0.0 (clang-1200.0.32.29)]
smart_open 5.1.0
boto3 1.17.93
moto 2.2.1