adds aws_sigv4 signer for s3 requests, adds python s3 credentials

[MatthewAitken]

• Adds aws sigv4 signer for fetch requests against private s3 buckets
• Implements neuroglancer python s3 credential provider using default boto3 credential chain
• Updates gcs kvstore to use python gcs credential provider
• Updates s3 kvstore to use python s3 credential provider

Todo:

• Add aws ngauth/cognito server
• maybe parse/validate aws service = "s3" from signer instead of hardcoding?
• get bucket name from url, get region from HeadBucket request? instead of from credentials
• clean up applyCredentials function

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[fcollman]

@jbms Matt is working on figuring out what the CLA signing situation is in general at the Allen, but we would appreciate any feedback you have on this sooner rather than later.

[jbms]

In general this looks fine --- I don't think there are many issues.

[bentaculum]

Hi @MatthewAitken, thanks for implementing this. May I ask you for an MWE to use this?

Here is what I tried, but I still get some permission error in the client:

AWS_PROFILE="my-profile" ipython, which handles passing specific credentials to a boto3 session that only reads the default ones.

Then in ipython

import neuroglancer
v = neuroglancer.Viewer()
with v.txn() as state:
    state.layers["test"] = neuroglancer.ImageLayer(source="s3://bucket/example.zarr/")
print(v)

[MatthewAitken]

Hi @MatthewAitken, thanks for implementing this. May I ask you for an MWE to use this?

Here is what I tried, but I still get some permission error in the client:

AWS_PROFILE="my-profile" ipython, which handles passing specific credentials to a boto3 session that only reads the default ones.

Then in ipython

import neuroglancer
v = neuroglancer.Viewer()
with v.txn() as state:
    state.layers["test"] = neuroglancer.ImageLayer(source="s3://bucket/example.zarr/")
print(v)

Apologies for the delay getting back to finish this.

You need to allow credentials (and replace mytoken with whatever token string you want)
viewer = neuroglancer.Viewer(allow_credentials=True, token="mytoken")