I run with `debug rgw = 10` and was able to find these lines at the end of a request to create the bucket.
Successfully creating a bucket with `bucket_location = US` looks like [1]this. Failing to create a bucket has "ERROR: S3 error: 400 (InvalidLocationConstraint): The specified location-constraint is not valid" on the CLI and [2]this (excerpt from the end of the request) in the rgw log (debug level 10). "create bucket location constraint" was not found in the log for successfully creating the bucket.
[1]
2018-02-26 19:52:36.419251 7f4bc9bc8700 10 cache put: name=local-atl.rgw.data.root++.bucket.meta.testerton:bef43c26-daf3-47ef-a3a5-e1167e3f88ac.39099765.1 info.flags=0x17
2018-02-26 19:52:36.419262 7f4bc9bc8700 10 adding local-atl.rgw.data.root++.bucket.meta.testerton:bef43c26-daf3-47ef-a3a5-e1167e3f88ac.39099765.1 to cache LRU end
2018-02-26 19:52:36.419266 7f4bc9bc8700 10 updating xattr: name=user.rgw.acl bl.length()=141
2018-02-26 19:52:36.423863 7f4bc9bc8700 10 RGWWatcher::handle_notify() notify_id 344855809097728 cookie 139963970426880 notifier 39099765 bl.length()=361
2018-02-26 19:52:36.423875 7f4bc9bc8700 10 cache put: name=local-atl.rgw.data.root++testerton info.flags=0x17
2018-02-26 19:52:36.423882 7f4bc9bc8700 10 adding local-atl.rgw.data.root++testerton to cache LRU end
[2]
2018-02-26 19:43:37.340289 7f466bbca700 2 req 428078:0.004204:s3:PUT /testraint/:create_bucket:executing
2018-02-26 19:43:37.340366 7f466bbca700 5 NOTICE: call to do_aws4_auth_completion
2018-02-26 19:43:37.340472 7f466bbca700 10 v4 auth ok -- do_aws4_auth_completion
2018-02-26 19:43:37.340715 7f466bbca700 10 create bucket location constraint: cn
2018-02-26 19:43:37.340766 7f466bbca700 0 location constraint (cn) can't be found.
2018-02-26 19:43:37.340794 7f466bbca700 2 req 428078:0.004701:s3:PUT /testraint/:create_bucket:completing
2018-02-26 19:43:37.341782 7f466bbca700 2 req 428078:0.005689:s3:PUT /testraint/:create_bucket:op status=-2208
2018-02-26 19:43:37.341792 7f466bbca700 2 req 428078:0.005707:s3:PUT /testraint/:create_bucket:http status=400
On Mon, Feb 26, 2018 at 2:36 PM Yehuda Sadeh-Weinraub <yehuda@xxxxxxxxxx> wrote:
I'm not sure if the rgw logs (debug rgw = 20) specify explicitly why a
bucket creation is rejected in these cases, but it might be worth
trying to look at these. If not, then a tcpdump of the specific failed
request might shed some light (would be interesting to look at the
generated LocationConstraint).
Yehuda
On Mon, Feb 26, 2018 at 11:29 AM, David Turner <drakonstein@xxxxxxxxx> wrote:
> Our problem only appeared to be present in bucket creation. Listing,
> putting, etc objects in a bucket work just fine regardless of the
> bucket_location setting. I ran this test on a few different realms to see
> what would happen and only 1 of them had a problem. There isn't an obvious
> thing that steps out about it. The 2 local realms do not have multi-site,
> the internal realm has multi-site and the operations were performed on the
> primary zone for the zonegroup.
>
> Worked with non 'US' bucket_location for s3cmd to create bucket:
> realm=internal
> zonegroup=internal-ga
> zone=internal-atl
>
> Failed with non 'US' bucket_location for s3cmd to create bucket:
> realm=local-atl
> zonegroup=local-atl
> zone=local-atl
>
> Worked with non 'US' bucket_location for s3cmd to create bucket:
> realm=local
> zonegroup=local
> zone=local
>
> I was thinking it might have to do with all of the parts being named the
> same, but I made sure to do the last test to confirm. Interestingly it's
> only bucket creation that has a problem and it's fine as long as I put 'US'
> as the bucket_location.
>
> On Mon, Feb 19, 2018 at 6:48 PM F21 <f21.groups@xxxxxxxxx> wrote:
>>
>> I am using the official ceph/daemon docker image. It starts RGW and
>> creates a zonegroup and zone with their names set to an empty string:
>>
>> https://github.com/ceph/ceph-container/blob/master/ceph-releases/luminous/ubuntu/16.04/daemon/start_rgw.sh#L36:54
>>
>> $RGW_ZONEGROUP and $RGW_ZONE are both empty strings by default:
>>
>> https://github.com/ceph/ceph-container/blob/master/ceph-releases/luminous/ubuntu/16.04/daemon/variables_entrypoint.sh#L46
>>
>> Here's what I get when I query RGW:
>>
>> $ radosgw-admin zonegroup list
>> {
>> "default_info": "",
>> "zonegroups": [
>> "default"
>> ]
>> }
>>
>> $ radosgw-admin zone list
>> {
>> "default_info": "",
>> "zones": [
>> "default"
>> ]
>> }
>>
>> On 20/02/2018 10:33 AM, Yehuda Sadeh-Weinraub wrote:
>> > What is the name of your zonegroup?
>> >
>> > On Mon, Feb 19, 2018 at 3:29 PM, F21 <f21.groups@xxxxxxxxx> wrote:
>> >> I've done some debugging and the LocationConstraint is not being set by
>> >> the
>> >> SDK by default.
>> >>
>> >> I do, however, need to set the region on the client to us-east-1 for it
>> >> to
>> >> work. Anything else will return an InvalidLocationConstraint error.
>> >>
>> >> Francis
>> >>
>> >>
>> >> On 20/02/2018 8:40 AM, Yehuda Sadeh-Weinraub wrote:
>> >>> Sounds like the go sdk adds a location constraint to requests that
>> >>> don't go to us-east-1. RGW itself is definitely isn't tied to
>> >>> us-east-1, and does not know anything about it (unless you happen to
>> >>> have a zonegroup named us-east-1). Maybe there's a way to configure
>> >>> the sdk to avoid doing that?
>> >>>
>> >>> Yehuda
>> >>>
>> >>> On Sun, Feb 18, 2018 at 1:54 PM, F21 <f21.groups@xxxxxxxxx> wrote:
>> >>>> I am using the AWS Go SDK v2 (https://github.com/aws/aws-sdk-go-v2)
>> >>>> to
>> >>>> talk
>> >>>> to my RGW instance using the s3 interface. I am running ceph in
>> >>>> docker
>> >>>> using
>> >>>> the ceph/daemon docker images in demo mode. The RGW is started with a
>> >>>> zonegroup and zone with their names set to an empty string by the
>> >>>> scripts
>> >>>> in
>> >>>> the image.
>> >>>>
>> >>>> I have ForcePathStyle for the client set to true, because I want to
>> >>>> access
>> >>>> all my buckets using the path: myrgw.instance:8080/somebucket.
>> >>>>
>> >>>> I noticed that if I set the region for the client to anything other
>> >>>> than
>> >>>> us-east-1, I get this error when creating a bucket:
>> >>>> InvalidLocationConstraint: The specified location-constraint is not
>> >>>> valid.
>> >>>>
>> >>>> If I set the region in the client to something made up, such as
>> >>>> "ceph"
>> >>>> and
>> >>>> the LocationConstraint to "ceph", I still get the same error.
>> >>>>
>> >>>> The only way to get my buckets to create successfully is to set the
>> >>>> client's
>> >>>> region to us-east-1. I have grepped the ceph code base and cannot
>> >>>> find
>> >>>> any
>> >>>> references to us-east-1. In addition, I looked at the AWS docs for
>> >>>> calculating v4 signatures and us-east-1 is the default region but I
>> >>>> can
>> >>>> see
>> >>>> that the region string is used in the calculation (i.e. the region is
>> >>>> not
>> >>>> ignored when calculating the signature if it is set to us-east-1).
>> >>>>
>> >>>> Why do my buckets create successfully if I set the region in my s3
>> >>>> client
>> >>>> to
>> >>>> us-east-1, but not otherwise? If I do not want to use us-east-1 as my
>> >>>> default region, for example, if I want us-west-1 as my default
>> >>>> region,
>> >>>> what
>> >>>> should I be configuring in ceph?
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Francis
>> >>>>
>> >>>> _______________________________________________
>> >>>> ceph-users mailing list
>> >>>> ceph-users@xxxxxxxxxxxxxx
>> >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >>
>> >>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
