On 9/18/19 4:27 AM, Yuval Lifshitz wrote:
On Mon, Sep 16, 2019 at 6:56 PM Casey Bodley <cbodley@xxxxxxxxxx
<mailto:cbodley@xxxxxxxxxx>> wrote:
That's so cool! We've been struggling to find ways to expose s3
extensions, and the client-side validation of clients like boto
has made
this really difficult.
I'm curious how granular we can make these extensions. For
example, if
we wanted to expose our ?allow-unordered extension to the ListObjects
op, would we have to replace the entire
botocore/data/s3/2006-03-01/service-2.json file? Or could we
provide a
new json file that only specifies a modified ListObjectsRequest
structure? If we can make them that granular, it would limit
conflicts
with upstream boto and be much easier to maintain. It could also
make it
easy for our users to opt-in to certain extensions.
- tried adding a different file, only with the changed structures, but
boto3 ignored it
The comment about 'sdk-extras' in
https://github.com/boto/botocore/blob/develop/botocore/loaders.py#L98-L102
sounds promising for this.
For example, I see a botocore/data/neptune/2014-10-31/service-2.json and
a service-2.sdk-extras.json that 'merges' an extra SourceRegion string
into two message types defined in service-2.json. That sounds like
exactly what we want; it may just be a matter of naming the files
correctly, ie 'service-2.*.json'.
- tried using the same file, but including only the
modified structures - in this case boto3 took it but failed to find
all the other definitions
I think the best option would be to have our own version of the file,
and regularly rebase it from botocore.
This services-2.json file changes about once a month (see [3]), so, it
should not be too bad to track it.
Another option would be to change the loaders code to allow for
merging the content from multiple json files, but that would be a
bigger task, IMO.
Exploring this for s3tests/pubsub in teuthology sounds like a good
first
step!
where do you think we should put that file?
If you want to start by testing extensions to s3 notifications, the
pubsub test driver in qa/tasks/rgw_multisite_tests.py could install a
file under /home/ubuntu/.aws/models/.
On 9/16/19 11:16 AM, Yuval Lifshitz wrote:
> Hi All,
> Recently I added a support for bucket notifications, as part of the
> feature the standard (s3) way for filtering notification was
extended
> with new fields that are not supported by AWS. This poses 2 issues:
> - how to test it
> - how users should use it
> Crafting these messages "by hand" (e.g. by using curl) is not ideal
> for both usecases.
>
> However, according to this [1], there is an easy way to extend the
> standard by changing a json file and then placing it under
> ~/.aws/models/s3/...
> Or, placing it anywhere else, and then setting its path first in
> the AWS_DATA_PATH search path (I tried the 1st options and it
worked!).
>
> I guess that this is not the only case where we want to
extend/modify
> what ASW/boto3 supports.
> Would suggest that we add our own versions of the relevant files
> (e.g.[2]) into the ceph repo (not sure where?).
> This could be used for:
> - teuthology infra can take these files and use them, so that the
> clients could test our extension natively with boto3
> - we can add documentation around these files, so they are
available
> to users (maybe deliver them in the installation packages)
>
> Of course, we will have to track changes to the original file, and
> merge from time top time...
>
> Yuval
>
> [1]
https://github.com/boto/botocore/blob/develop/botocore/loaders.py#L33
> [2]
>
https://github.com/boto/botocore/blob/develop/botocore/data/s3/2006-03-01/service-2.json
>
>
> _______________________________________________
> Dev mailing list -- dev@xxxxxxx <mailto:dev@xxxxxxx>
> To unsubscribe send an email to dev-leave@xxxxxxx
<mailto:dev-leave@xxxxxxx>
_______________________________________________
Dev mailing list -- dev@xxxxxxx <mailto:dev@xxxxxxx>
To unsubscribe send an email to dev-leave@xxxxxxx
<mailto:dev-leave@xxxxxxx>
[3]
https://github.com/boto/botocore/commits/develop/botocore/data/s3/2006-03-01/service-2.json
_______________________________________________
Dev mailing list -- dev@xxxxxxx
To unsubscribe send an email to dev-leave@xxxxxxx
