On Mon, Sep 16, 2019 at 6:56 PM Casey Bodley <cbodley@xxxxxxxxxx> wrote:
That's so cool! We've been struggling to find ways to expose s3
extensions, and the client-side validation of clients like boto has made
this really difficult.
I'm curious how granular we can make these extensions. For example, if
we wanted to expose our ?allow-unordered extension to the ListObjects
op, would we have to replace the entire
botocore/data/s3/2006-03-01/service-2.json file? Or could we provide a
new json file that only specifies a modified ListObjectsRequest
structure? If we can make them that granular, it would limit conflicts
with upstream boto and be much easier to maintain. It could also make it
easy for our users to opt-in to certain extensions.
- tried adding a different file, only with the changed structures, but boto3 ignored it
- tried using the same file, but including only the modified structures - in this case boto3 took it but failed to find all the other definitions
I think the best option would be to have our own version of the file, and regularly rebase it from botocore.
This services-2.json file changes about once a month (see [3]), so, it should not be too bad to track it.
Another option would be to change the loaders code to allow for merging the content from multiple json files, but that would be a bigger task, IMO.
Exploring this for s3tests/pubsub in teuthology sounds like a good first
step!
where do you think we should put that file?
On 9/16/19 11:16 AM, Yuval Lifshitz wrote:
> Hi All,
> Recently I added a support for bucket notifications, as part of the
> feature the standard (s3) way for filtering notification was extended
> with new fields that are not supported by AWS. This poses 2 issues:
> - how to test it
> - how users should use it
> Crafting these messages "by hand" (e.g. by using curl) is not ideal
> for both usecases.
>
> However, according to this [1], there is an easy way to extend the
> standard by changing a json file and then placing it under
> ~/.aws/models/s3/...
> Or, placing it anywhere else, and then setting its path first in
> the AWS_DATA_PATH search path (I tried the 1st options and it worked!).
>
> I guess that this is not the only case where we want to extend/modify
> what ASW/boto3 supports.
> Would suggest that we add our own versions of the relevant files
> (e.g.[2]) into the ceph repo (not sure where?).
> This could be used for:
> - teuthology infra can take these files and use them, so that the
> clients could test our extension natively with boto3
> - we can add documentation around these files, so they are available
> to users (maybe deliver them in the installation packages)
>
> Of course, we will have to track changes to the original file, and
> merge from time top time...
>
> Yuval
>
> [1] https://github.com/boto/botocore/blob/develop/botocore/loaders.py#L33
> [2]
> https://github.com/boto/botocore/blob/develop/botocore/data/s3/2006-03-01/service-2.json
>
>
> _______________________________________________
> Dev mailing list -- dev@xxxxxxx
> To unsubscribe send an email to dev-leave@xxxxxxx
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
