Okay, will do. If the problem goes away with filestore, I'll switch back to bluestore again and re-duplicate the problem. In that case, are there particular things you would like me to collect? Or clues I should look for in logs? Thanks, Kevan On 5/24/16, 4:06 PM, "Samuel Just" <sjust@xxxxxxxxxx> wrote: >My money is on bluestore. If you can try to reproduce on filestore, >that would rapidly narrow it down. >-Sam > >On Tue, May 24, 2016 at 1:53 PM, Kevan Rehm <krehm@xxxxxxxx> wrote: >> Nope, not using tiering. >> >> Also, this is my second attempt, this is repeatable for me, I'm trying >>to >> duplicate a previous occurrence of this same problem to collect useful >> debug data. In the previous case, I was eventually able to get rid of >>the >> objects (but have forgotten how), but that was followed by 22 of the 24 >> OSDs crashing hard. Took me quite a while to re-deploy and get it >>working >> again. I want to get a small, repeatable example for the Ceph guys to >> look at, assuming it's a bug. >> >> Don't know if it's related to the bluestore OSDs or not, still getting >>my >> feet wet with Ceph. >> >> Kevan >> >> On 5/24/16, 3:47 PM, "Jason Dillaman" <jdillama@xxxxxxxxxx> wrote: >> >>>Any chance you are using cache tiering? It's odd that you can see the >>>objects through "rados ls" but cannot delete them with "rados rm". >>> >>>On Tue, May 24, 2016 at 4:34 PM, Kevan Rehm <krehm@xxxxxxxx> wrote: >>>> Greetings, >>>> >>>> I have a small Ceph 10.2.1 test cluster using a 3-replicate pool based >>>>on 24 >>>> SSDs configured with bluestore. I created and wrote an rbd image >>>>called >>>> "image1", then deleted the image again. >>>> >>>> rbd -p ssd_replica create --size 100G image1 >>>> >>>> rbd --pool ssd_replica bench-write --io-size 2M --io-threads 16 >>>>--io-total >>>> 100G --io-pattern seq image1 >>>> >>>> rbd -p ssd_replica rm image1 >>>> >>>> >>>> The rbd rm command completed successfully, but not all the >>>>image-related >>>> files disappeared from the pool. The pool still contains files: >>>> >>>> >>>> rbd_directory >>>> >>>> rbd_id.image1 >>>> >>>> rbd_object_map.15ed238e1f29 >>>> >>>> >>>> plus 2938 other objects of the form >>>> "rbd_data.15ed238e1f29.000000000000xxxx". >>>> >>>> Attempting to re-delete image1 does not work. Attempting to directly >>>>delete >>>> object rbd_id.image1 and one of the data objects doesn't work either. >>>>The >>>> commands all fail with "No such file or directory" yet they are >>>>obviously >>>> there. (Warning messages stripped for brevity.) >>>> >>>> [root@alpha1-p200 fio]# rbd -p ssd_replica rm image1 >>>> >>>> Removing image: 0% complete...failed. >>>> >>>> rbd: delete error: (2) No such file or directory >>>> >>>> [root@alpha1-p200 fio]# rados -p ssd_replica rm rbd_id.image1 >>>> >>>> error removing ssd_replica>rbd_id.image1: (2) No such file or >>>>directory >>>> >>>> [root@alpha1-p200 fio]# rados -p ssd_replica rm >>>> rbd_data.15ed238e1f29.0000000000005b56 >>>> >>>> error removing ssd_replica>rbd_data.15ed238e1f29.0000000000005b56: (2) >>>>No >>>> such file or directory >>>> >>>> >>>> Is this a bug, and if so, where should I look for the cause? Anyone >>>>know >>>> how to delete these objects? >>>> >>>> Thanks, Kevan >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>> >>> >>> >>>-- >>>Jason >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |