Nope, not using tiering. Also, this is my second attempt, this is repeatable for me, I'm trying to duplicate a previous occurrence of this same problem to collect useful debug data. In the previous case, I was eventually able to get rid of the objects (but have forgotten how), but that was followed by 22 of the 24 OSDs crashing hard. Took me quite a while to re-deploy and get it working again. I want to get a small, repeatable example for the Ceph guys to look at, assuming it's a bug. Don't know if it's related to the bluestore OSDs or not, still getting my feet wet with Ceph. Kevan On 5/24/16, 3:47 PM, "Jason Dillaman" <jdillama@xxxxxxxxxx> wrote: >Any chance you are using cache tiering? It's odd that you can see the >objects through "rados ls" but cannot delete them with "rados rm". > >On Tue, May 24, 2016 at 4:34 PM, Kevan Rehm <krehm@xxxxxxxx> wrote: >> Greetings, >> >> I have a small Ceph 10.2.1 test cluster using a 3-replicate pool based >>on 24 >> SSDs configured with bluestore. I created and wrote an rbd image called >> "image1", then deleted the image again. >> >> rbd -p ssd_replica create --size 100G image1 >> >> rbd --pool ssd_replica bench-write --io-size 2M --io-threads 16 >>--io-total >> 100G --io-pattern seq image1 >> >> rbd -p ssd_replica rm image1 >> >> >> The rbd rm command completed successfully, but not all the image-related >> files disappeared from the pool. The pool still contains files: >> >> >> rbd_directory >> >> rbd_id.image1 >> >> rbd_object_map.15ed238e1f29 >> >> >> plus 2938 other objects of the form >> "rbd_data.15ed238e1f29.000000000000xxxx". >> >> Attempting to re-delete image1 does not work. Attempting to directly >>delete >> object rbd_id.image1 and one of the data objects doesn't work either. >>The >> commands all fail with "No such file or directory" yet they are >>obviously >> there. (Warning messages stripped for brevity.) >> >> [root@alpha1-p200 fio]# rbd -p ssd_replica rm image1 >> >> Removing image: 0% complete...failed. >> >> rbd: delete error: (2) No such file or directory >> >> [root@alpha1-p200 fio]# rados -p ssd_replica rm rbd_id.image1 >> >> error removing ssd_replica>rbd_id.image1: (2) No such file or directory >> >> [root@alpha1-p200 fio]# rados -p ssd_replica rm >> rbd_data.15ed238e1f29.0000000000005b56 >> >> error removing ssd_replica>rbd_data.15ed238e1f29.0000000000005b56: (2) >>No >> such file or directory >> >> >> Is this a bug, and if so, where should I look for the cause? Anyone >>know >> how to delete these objects? >> >> Thanks, Kevan >> >> >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > >-- >Jason _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |