> On Jan 4, 2017, at 10:12 AM, Orit Wasserman <owasserm@xxxxxxxxxx> wrote:
>
> On Wed, Jan 4, 2017 at 7:08 PM, Brian Andrus <brian.andrus@xxxxxxxxxxxxx> wrote:
>> Regardless of whether it worked before, have you verified your RadosGWs have
>> write access to monitors? They will need it if you want the RadosGW to
>> create its own pools.
>>
>> ceph auth get <client.radosgwID>
>>
>
> I agree, it could be permissions issue
# ceph auth get client.radosgw.gateway
exported keyring for client.radosgw.gateway
[client.radosgw.gateway]
key = XXXXXXXXXX
caps mon = "allow *"
caps osd = "allow *”
That is not the issue. I have a strong feeling that the issue is with the default CRUSH rule, which is used for creating the RGW pools automatically. I will verify and update.
Thanks,
Nitin
>
>> On Wed, Jan 4, 2017 at 8:59 AM, Kamble, Nitin A <Nitin.Kamble@xxxxxxxxxxxx>
>> wrote:
>>>
>>>
>>>> On Dec 26, 2016, at 2:48 AM, Orit Wasserman <owasserm@xxxxxxxxxx> wrote:
>>>>
>>>> On Fri, Dec 23, 2016 at 3:42 AM, Kamble, Nitin A
>>>> <Nitin.Kamble@xxxxxxxxxxxx> wrote:
>>>>> I am trying to setup radosgw on a ceph cluster, and I am seeing some
>>>>> issues where google is not helping. I hope some of the developers would be
>>>>> able to help here.
>>>>>
>>>>>
>>>>> I tried to create radosgw as mentioned here [0] on a jewel cluster. And
>>>>> it gives the following error in log file after starting radosgw.
>>>>>
>>>>>
>>>>> 2016-12-22 17:36:46.755786 7f084beeb9c0 0 set uid:gid to 167:167
>>>>> (ceph:ceph)
>>>>> 2016-12-22 17:36:46.755849 7f084beeb9c0 0 ceph version
>>>>> 10.2.2-118-g894a5f8 (894a5f8d878d4b267f80b90a4bffce157f2b4ba7), process
>>>>> radosgw, pid 10092
>>>>> 2016-12-22 17:36:46.763821 7f084beeb9c0 1 -- :/0 messenger.start
>>>>> 2016-12-22 17:36:46.764731 7f084beeb9c0 1 -- :/1011033520 -->
>>>>> 39.0.16.7:6789/0 -- auth(proto 0 40 bytes epoch 0) v1 -- ?+0 0x7f084c8e9f60
>>>>> con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.765055 7f084beda700 1 -- 39.0.16.9:0/1011033520
>>>>> learned my addr 39.0.16.9:0/1011033520
>>>>> 2016-12-22 17:36:46.765492 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 1 ==== mon_map magic: 0 v1 ==== 195+0+0
>>>>> (146652916 0 0) 0x7f0814000a60 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.765562 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ====
>>>>> 33+0+0 (1206278719 0 0) 0x7f0814000ee0 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.765697 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0
>>>>> 0x7f08180013b0 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.765968 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ====
>>>>> 222+0+0 (4230455906 0 0) 0x7f0814000ee0 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.766053 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- ?+0
>>>>> 0x7f0818001830 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.766315 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ====
>>>>> 425+0+0 (3179848142 0 0) 0x7f0814001180 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.766383 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x7f084c8ea440
>>>>> con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.766452 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x7f084c8ea440
>>>>> con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.766518 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 195+0+0
>>>>> (146652916 0 0) 0x7f0814001110 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.766671 7f08227fc700 2
>>>>> RGWDataChangesLog::ChangesRenewThread: start
>>>>> 2016-12-22 17:36:46.766691 7f084beeb9c0 20 get_system_obj_state:
>>>>> rctx=0x7ffec2850d00 obj=.rgw.root:default.realm state=0x7f084c8efdf8
>>>>> s->prefetch_data=0
>>>>> 2016-12-22 17:36:46.766750 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 6 ==== osd_map(9506..9506 src has 8863..9506) v3
>>>>> ==== 66915+0+0 (689048617 0 0) 0x7f0814011680 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.767029 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0
>>>>> 0x7f084c8f05f0 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.767163 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 7 ==== mon_get_version_reply(handle=1
>>>>> version=9506) v2 ==== 24+0+0 (2817198406 0 0) 0x7f0814001110 con
>>>>> 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.767214 7f084beeb9c0 20 get_system_obj_state:
>>>>> rctx=0x7ffec2850210 obj=.rgw.root:default.realm state=0x7f084c8efdf8
>>>>> s->prefetch_data=0
>>>>> 2016-12-22 17:36:46.767231 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- mon_get_version(what=osdmap handle=2) v1 -- ?+0
>>>>> 0x7f084c8f0ac0 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.767341 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 8 ==== mon_get_version_reply(handle=2
>>>>> version=9506) v2 ==== 24+0+0 (1826043941 0 0) 0x7f0814001110 con
>>>>> 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.767367 7f084beeb9c0 10 could not read realm id: (2)
>>>>> No such file or directory
>>>>> 2016-12-22 17:36:46.767390 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- mon_get_version(what=osdmap handle=3) v1 -- ?+0
>>>>> 0x7f084c8efe50 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.767496 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 9 ==== mon_get_version_reply(handle=3
>>>>> version=9506) v2 ==== 24+0+0 (3600349867 0 0) 0x7f0814001110 con
>>>>> 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.767518 7f084beeb9c0 10 failed to list objects
>>>>> pool_iterate_begin() returned r=-2
>>>>> 2016-12-22 17:36:46.767542 7f084beeb9c0 20 get_system_obj_state:
>>>>> rctx=0x7ffec2850420 obj=.rgw.root:zone_names.default state=0x7f084c8f0f38
>>>>> s->prefetch_data=0
>>>>> 2016-12-22 17:36:46.767554 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- mon_get_version(what=osdmap handle=4) v1 -- ?+0
>>>>> 0x7f084c8f1630 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.767660 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 10 ==== mon_get_version_reply(handle=4
>>>>> version=9506) v2 ==== 24+0+0 (4282592274 0 0) 0x7f0814001110 con
>>>>> 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.767685 7f084beeb9c0 0 error in read_id for id :
>>>>> (2) No such file or directory
>>>>> 2016-12-22 17:36:46.767700 7f084beeb9c0 20 get_system_obj_state:
>>>>> rctx=0x7ffec2850ed0 obj=.rgw.root:region_map state=0x7f084c8f0f38
>>>>> s->prefetch_data=0
>>>>> 2016-12-22 17:36:46.767715 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- mon_get_version(what=osdmap handle=5) v1 -- ?+0
>>>>> 0x7f084c8f1c10 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.767830 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 11 ==== mon_get_version_reply(handle=5
>>>>> version=9506) v2 ==== 24+0+0 (1158475420 0 0) 0x7f0814001110 con
>>>>> 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.767855 7f084beeb9c0 10 cannot find current period
>>>>> zonegroup using local zonegroup
>>>>> 2016-12-22 17:36:46.767868 7f084beeb9c0 20 get_system_obj_state:
>>>>> rctx=0x7ffec2850880 obj=.rgw.root:default.realm state=0x7f084c8f0f38
>>>>> s->prefetch_data=0
>>>>> 2016-12-22 17:36:46.767880 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- mon_get_version(what=osdmap handle=6) v1 -- ?+0
>>>>> 0x7f084c8f21f0 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.767983 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 12 ==== mon_get_version_reply(handle=6
>>>>> version=9506) v2 ==== 24+0+0 (2385567743 0 0) 0x7f0814001110 con
>>>>> 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.768007 7f084beeb9c0 10 could not read realm id: (2)
>>>>> No such file or directory
>>>>> 2016-12-22 17:36:46.768014 7f084beeb9c0 10 Creating default zonegroup
>>>>> 2016-12-22 17:36:46.768034 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- mon_get_version(what=osdmap handle=7) v1 -- ?+0
>>>>> 0x7f084c8f1080 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.768142 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 13 ==== mon_get_version_reply(handle=7
>>>>> version=9506) v2 ==== 24+0+0 (880745841 0 0) 0x7f0814001110 con
>>>>> 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.768167 7f084beeb9c0 10 couldn't find old data
>>>>> placement pools config, setting up new ones for the zone
>>>>> 2016-12-22 17:36:46.768190 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- mon_get_version(what=osdmap handle=8) v1 -- ?+0
>>>>> 0x7f084c8f26c0 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.768293 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 14 ==== mon_get_version_reply(handle=8
>>>>> version=9506) v2 ==== 24+0+0 (3716641421 0 0) 0x7f0814001110 con
>>>>> 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.768314 7f084beeb9c0 10 failed to list objects
>>>>> pool_iterate_begin() returned r=-2
>>>>> 2016-12-22 17:36:46.768320 7f084beeb9c0 10 WARNING: store->list_zones()
>>>>> returned r=-2
>>>>> 2016-12-22 17:36:46.768346 7f084beeb9c0 20 get_system_obj_state:
>>>>> rctx=0x7ffec2850400 obj=.rgw.root:zone_names.default state=0x7f084c8f2b38
>>>>> s->prefetch_data=0
>>>>> 2016-12-22 17:36:46.768362 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- mon_get_version(what=osdmap handle=9) v1 -- ?+0
>>>>> 0x7f084c8f3140 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.768463 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 15 ==== mon_get_version_reply(handle=9
>>>>> version=9506) v2 ==== 24+0+0 (1741205507 0 0) 0x7f0814001110 con
>>>>> 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.768578 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- mon_get_version(what=osdmap handle=10) v1 -- ?+0
>>>>> 0x7f084c8f2aa0 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.768681 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 16 ==== mon_get_version_reply(handle=10
>>>>> version=9506) v2 ==== 24+0+0 (2901705056 0 0) 0x7f0814001110 con
>>>>> 0x7f084c8e9480
>>>>> 2016-12-22 17:36:46.768724 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- pool_op(create pool 0 auid 0 tid 1 name .rgw.root
>>>>> v0) v4 -- ?+0 0x7f084c8f4520 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:47.052029 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 17 ==== pool_op_reply(tid 1 (34) Numerical result
>>>>> out of range v9507) v1 ==== 43+0+0 (366377631 0 0) 0x7f0814001110 con
>>>>> 0x7f084c8e9480
>>>>
>>>
>>> Thanks for the response Orit.
>>>
>>>> what filesystem are you using? we longer support ext4
>>>
>>> OSDs are using XFS for Filestore.
>>>
>>>> Another option is version mismatch between rgw and the ods.
>>>
>>> Exact same version of ceph binaries are installed on OSD, MON and RGW
>>> nodes.
>>>
>>> Is there anything useful in the error messages?
>>>
>>> 2016-12-22 17:36:46.768314 7f084beeb9c0 10 failed to list objects
>>> pool_iterate_begin() returned r=-2
>>> 2016-12-22 17:36:46.768320 7f084beeb9c0 10 WARNING: store->list_zones()
>>> returned r=-2
>>>
>>> Is this the point where the failure has begun?
>>>
>>> As I see, the basic issue is, RGW is not able to create the needed pools
>>> on demand. I wish there was more detailed output regarding the Numerical
>>> result out of range issue.
>>> I am suspecting it may be related to the set of defaults used while
>>> creating pools automatically. Possibly the default crush rule.
>>>
>>> Thanks,
>>> Nitin
>>>
>>>>
>>>> Orit
>>>>
>>>>> 2016-12-22 17:36:47.052067 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> --> 39.0.16.7:6789/0 -- mon_subscribe({osdmap=9507}) v2 -- ?+0
>>>>> 0x7f0818022bb0 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:47.055809 7f082a7fc700 1 -- 39.0.16.9:0/1011033520
>>>>> <== mon.0 39.0.16.7:6789/0 18 ==== osd_map(9507..9507 src has 8863..9507) v3
>>>>> ==== 214+0+0 (1829214220 0 0) 0x7f0814001110 con 0x7f084c8e9480
>>>>> 2016-12-22 17:36:47.055858 7f084beeb9c0 0 ERROR: storing info for
>>>>> 84f3cdd9-71d9-4d74-a6ba-c0e87d776a2b: (34) Numerical result out of range
>>>>> 2016-12-22 17:36:47.055869 7f084beeb9c0 0 create_default: error in
>>>>> create_default zone params: (34) Numerical result out of range
>>>>> 2016-12-22 17:36:47.055876 7f084beeb9c0 0 failure in zonegroup
>>>>> create_default: ret -34 (34) Numerical result out of range
>>>>> 2016-12-22 17:36:47.055970 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> mark_down 0x7f084c8e9480 -- 0x7f084c8ec0f0
>>>>> 2016-12-22 17:36:47.056169 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> mark_down_all
>>>>> 2016-12-22 17:36:47.056263 7f084beeb9c0 1 -- 39.0.16.9:0/1011033520
>>>>> shutdown complete.
>>>>> 2016-12-22 17:36:47.056426 7f084beeb9c0 -1 Couldn't init storage
>>>>> provider (RADOS)
>>>>>
>>>>>
>>>>>
>>>>> I did not create the pools for rgw, as they get created automatically.
>>>>> few weeks back, I could setup RGW on jewel successfully. But this time I am
>>>>> not able to see any obvious issues which I can fix.
>>>>>
>>>>>
>>>>> [0] http://docs.ceph.com/docs/jewel/radosgw/config/
>>>>>
>>>>> Thanks in advance,
>>>>> Nitin
>>>>>
>>>>> _______________________________________________
>>>>> ceph-users mailing list
>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@xxxxxxxxxxxxxx
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>>
>> --
>> Brian Andrus
>> Cloud Systems Engineer
>> DreamHost, LLC
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
