to anyone interested, I have a PR for review: https://github.com/ceph/ceph/pull/30600
once this is merged, more extensions (exiting or new ones) could be added to the same place.
On Wed, Sep 18, 2019 at 6:21 PM Casey Bodley <cbodley@xxxxxxxxxx> wrote:
I managed to hack up support for our ?allow-unordered extension to s3
object listing, by putting this file [1] in
~/.aws/models/s3/2006-03-01/service-2.sdk-extras.json. With that, a
boto3 script [2] was able to call client.list_objects(Bucket=bucketname,
AllowUnordered=True), and the ?allow-unordered=true param shows up in
the radosgw log!
It looks like the name 'service-2.sdk-extras.json' is the only one that
will work automatically, so while we -can- separate our extensions from
the upstream boto project, we do still need to bundle them all in the
same file. And, I guess, hope that botocore doesn't add a conflicting
service-2.sdk-extras.json file for s3 at some point in the future.
[1] https://gist.github.com/cbodley/eaf483f1e372224b130e15757d9e5634
[2] https://gist.github.com/cbodley/7ecbbaf4d220a120b2a893a4abf7ba0d
On 9/18/19 10:20 AM, Casey Bodley wrote:
>
> On 9/18/19 4:27 AM, Yuval Lifshitz wrote:
>>
>> On Mon, Sep 16, 2019 at 6:56 PM Casey Bodley <cbodley@xxxxxxxxxx
>> <mailto:cbodley@xxxxxxxxxx>> wrote:
>>
>> That's so cool! We've been struggling to find ways to expose s3
>> extensions, and the client-side validation of clients like boto
>> has made
>> this really difficult.
>>
>> I'm curious how granular we can make these extensions. For
>> example, if
>> we wanted to expose our ?allow-unordered extension to the
>> ListObjects
>> op, would we have to replace the entire
>> botocore/data/s3/2006-03-01/service-2.json file? Or could we
>> provide a
>> new json file that only specifies a modified ListObjectsRequest
>> structure? If we can make them that granular, it would limit
>> conflicts
>> with upstream boto and be much easier to maintain. It could also
>> make it
>> easy for our users to opt-in to certain extensions.
>>
>>
>> - tried adding a different file, only with the changed structures,
>> but boto3 ignored it
>
> The comment about 'sdk-extras' in
> https://github.com/boto/botocore/blob/develop/botocore/loaders.py#L98-L102
> sounds promising for this.
>
> For example, I see a botocore/data/neptune/2014-10-31/service-2.json
> and a service-2.sdk-extras.json that 'merges' an extra SourceRegion
> string into two message types defined in service-2.json. That sounds
> like exactly what we want; it may just be a matter of naming the files
> correctly, ie 'service-2.*.json'.
>
>
>> - tried using the same file, but including only the
>> modified structures - in this case boto3 took it but failed to find
>> all the other definitions
>>
>> I think the best option would be to have our own version of the file,
>> and regularly rebase it from botocore.
>> This services-2.json file changes about once a month (see [3]), so,
>> it should not be too bad to track it.
>> Another option would be to change the loaders code to allow for
>> merging the content from multiple json files, but that would be a
>> bigger task, IMO.
>>
>> Exploring this for s3tests/pubsub in teuthology sounds like a good
>> first
>> step!
>>
>>
>> where do you think we should put that file?
>
> If you want to start by testing extensions to s3 notifications, the
> pubsub test driver in qa/tasks/rgw_multisite_tests.py could install a
> file under /home/ubuntu/.aws/models/.
>
>
>> On 9/16/19 11:16 AM, Yuval Lifshitz wrote:
>> > Hi All,
>> > Recently I added a support for bucket notifications, as part of
>> the
>> > feature the standard (s3) way for filtering notification was
>> extended
>> > with new fields that are not supported by AWS. This poses 2
>> issues:
>> > - how to test it
>> > - how users should use it
>> > Crafting these messages "by hand" (e.g. by using curl) is not
>> ideal
>> > for both usecases.
>> >
>> > However, according to this [1], there is an easy way to extend the
>> > standard by changing a json file and then placing it under
>> > ~/.aws/models/s3/...
>> > Or, placing it anywhere else, and then setting its path first in
>> > the AWS_DATA_PATH search path (I tried the 1st options and it
>> worked!).
>> >
>> > I guess that this is not the only case where we want to
>> extend/modify
>> > what ASW/boto3 supports.
>> > Would suggest that we add our own versions of the relevant files
>> > (e.g.[2]) into the ceph repo (not sure where?).
>> > This could be used for:
>> > - teuthology infra can take these files and use them, so that the
>> > clients could test our extension natively with boto3
>> > - we can add documentation around these files, so they are
>> available
>> > to users (maybe deliver them in the installation packages)
>> >
>> > Of course, we will have to track changes to the original file, and
>> > merge from time top time...
>> >
>> > Yuval
>> >
>> > [1]
>> https://github.com/boto/botocore/blob/develop/botocore/loaders.py#L33
>> > [2]
>> >
>> https://github.com/boto/botocore/blob/develop/botocore/data/s3/2006-03-01/service-2.json
>> >
>> >
>> > _______________________________________________
>> > Dev mailing list -- dev@xxxxxxx <mailto:dev@xxxxxxx>
>> > To unsubscribe send an email to dev-leave@xxxxxxx
>> <mailto:dev-leave@xxxxxxx>
>> _______________________________________________
>> Dev mailing list -- dev@xxxxxxx <mailto:dev@xxxxxxx>
>> To unsubscribe send an email to dev-leave@xxxxxxx
>> <mailto:dev-leave@xxxxxxx>
>>
>>
>> [3]
>> https://github.com/boto/botocore/commits/develop/botocore/data/s3/2006-03-01/service-2.json
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
