On Jun 13, 2013, at 4:03 PM, Yehuda Sadeh <yehuda@xxxxxxxxxxx> wrote: > On Thu, Jun 13, 2013 at 3:01 PM, John Nielsen <lists@xxxxxxxxxxxx> wrote: >> On Jun 12, 2013, at 8:15 PM, Yehuda Sadeh <yehuda@xxxxxxxxxxx> wrote: >> >>> On Wed, Jun 12, 2013 at 2:43 PM, John Nielsen <lists@xxxxxxxxxxxx> wrote: >>>> >>>> With: >>>> caps osd = "allow x, allow pool .pubintent-log rwx, allow pool .publog rwx, allow pool .pubrgw rwx, allow pool .pubrgw.buckets rwx, allow pool .pubrgw.control rwx, allow pool .pubrgw.gc rwx, allow pool .pubusage rwx, allow pool .pubusers rwx, allow pool .pubusers.email rwx, allow pool .pubusers.swift rwx, allow pool .pubusers.uid rwx" >>>> >>> >>> Instead of adding 'class-write', you removed 'class-read'. You >>> probably need both. >> >> Is "x" not the same as "class-write class-read"? In any case, I see the same behavior with "class-write class-read". >> >>>> I get the same result: >>>> >>>> 2013-06-12 13:47:21.711904 7f9f53244820 0 ceph version 0.61.3 (92b1e398576d55df8e5888dd1a9545ed3fd99532), process radosgw, pid 6173 >>>> 2013-06-12 13:47:21.727924 7f9f53244820 20 get_obj_state: rctx=0x2b72690 obj=.pubrgw:zone_info state=0x2b763d8 s->prefetch_data=0 >>>> 2013-06-12 13:47:21.727945 7f9f53244820 10 cache get: name=.pubrgw+zone_info : miss >>>> 2013-06-12 13:47:21.731676 7f9f37fff700 2 garbage collection: start >>>> 2013-06-12 13:47:21.734877 7f9f37fff700 0 ERROR: garbage collection process() returned error r=-1 > > Try setting 'debug ms = 1', and see what operation fails. You may want > to correlate that to an osd log. Here is the first error. Looks like it is failing to read zone_info from pool 13, which is .pubrgw, the zone root pool. That can't be good. 2013-06-13 16:04:59.568213 7f298ada2820 1 -- 10.53.128.81:0/1009892 --> 10.53.128.12:6814/22684 -- osd_op(client.174996.0:1 zone_info [getxattrs,stat] 13.c5edb0aa e3086) v4 -- ?+0 0x2984510 con 0x2984180 2013-06-13 16:04:59.570830 7f29880d0700 1 -- 10.53.128.81:0/1009892 <== osd.22 10.53.128.12:6814/22684 1 ==== osd_op_reply(1 zone_info [getxattrs,stat] ack = -1 (Operation not permitted)) v4 ==== 150+0+0 (2873444189 0 0) 0x7f29680009c0 con 0x2984180 There are no logs from osd.22 in this period (but I didn't turn on any extra debugging either). Running this command (on the gateway) also fails with the reduced permissions: # radosgw-admin -n client.radosgw.public --rgw-zone-root-pool=.pubrgw zone info couldn't init storage provider So now the questions is why? Is there something wrong with my caps line? (It does include "allow pool .pubrgw rwx"). It is stored okay: # ceph auth export client.radosgw.public export auth(auid = [redacted] key=[redacted] with 2 caps) [client.radosgw.public] key = [redacted] caps mon = "allow r" caps osd = "allow class-read class-write, allow pool .pubintent-log rwx, allow pool .publog rwx, allow pool .pubrgw rwx, allow pool .pubrgw.buckets rwx, allow pool .pubrgw.control rwx, allow pool .pubrgw.gc rwx, allow pool .pubusage rwx, allow pool .pubusers rwx, allow pool .pubusers.email rwx, allow pool .pubusers.swift rwx, allow pool .pubusers.uid rwx" Incidentally, it looks like the rgw tries to carry on in spite of the above error with unwanted results: 2013-06-13 16:04:59.571593 7f296ffff700 2 garbage collection: start 2013-06-13 16:04:59.571680 7f298ada2820 1 -- 10.53.128.81:0/1009892 --> 10.53.128.12:6803/21182 -- osd_op(client.174996.0:2 notify.0 [create 0~0] 26.4322fa9f e3086) v4 -- ?+0 0x2985430 con 0x29850a0 2013-06-13 16:04:59.571806 7f296ffff700 1 -- 10.53.128.81:0/1009892 --> 10.53.128.10:6811/17357 -- osd_op(client.174996.0:3 gc.4 [call lock.lock] 26.370bf669 e3086) v4 -- ?+0 0x7f2964002640 con 0x7f2964002270 2013-06-13 16:04:59.574337 7f29880d0700 1 -- 10.53.128.81:0/1009892 <== osd.18 10.53.128.12:6803/21182 1 ==== osd_op_reply(2 notify.0 [create 0~0] ack = -1 (Operation not permitted)) v4 ==== 107+0+0 (1777541243 0 0) 0x7f29540009a0 con 0x29850a0 It looks like those ops reference pool 26 which shouldn't exist. However I at some point ended up with a pool 26 whose name was the empty string. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |