My money is on bluestore. If you can try to reproduce on filestore, that would rapidly narrow it down. -Sam On Tue, May 24, 2016 at 1:53 PM, Kevan Rehm <krehm@xxxxxxxx> wrote: > Nope, not using tiering. > > Also, this is my second attempt, this is repeatable for me, I'm trying to > duplicate a previous occurrence of this same problem to collect useful > debug data. In the previous case, I was eventually able to get rid of the > objects (but have forgotten how), but that was followed by 22 of the 24 > OSDs crashing hard. Took me quite a while to re-deploy and get it working > again. I want to get a small, repeatable example for the Ceph guys to > look at, assuming it's a bug. > > Don't know if it's related to the bluestore OSDs or not, still getting my > feet wet with Ceph. > > Kevan > > On 5/24/16, 3:47 PM, "Jason Dillaman" <jdillama@xxxxxxxxxx> wrote: > >>Any chance you are using cache tiering? It's odd that you can see the >>objects through "rados ls" but cannot delete them with "rados rm". >> >>On Tue, May 24, 2016 at 4:34 PM, Kevan Rehm <krehm@xxxxxxxx> wrote: >>> Greetings, >>> >>> I have a small Ceph 10.2.1 test cluster using a 3-replicate pool based >>>on 24 >>> SSDs configured with bluestore. I created and wrote an rbd image called >>> "image1", then deleted the image again. >>> >>> rbd -p ssd_replica create --size 100G image1 >>> >>> rbd --pool ssd_replica bench-write --io-size 2M --io-threads 16 >>>--io-total >>> 100G --io-pattern seq image1 >>> >>> rbd -p ssd_replica rm image1 >>> >>> >>> The rbd rm command completed successfully, but not all the image-related >>> files disappeared from the pool. The pool still contains files: >>> >>> >>> rbd_directory >>> >>> rbd_id.image1 >>> >>> rbd_object_map.15ed238e1f29 >>> >>> >>> plus 2938 other objects of the form >>> "rbd_data.15ed238e1f29.000000000000xxxx". >>> >>> Attempting to re-delete image1 does not work. Attempting to directly >>>delete >>> object rbd_id.image1 and one of the data objects doesn't work either. >>>The >>> commands all fail with "No such file or directory" yet they are >>>obviously >>> there. (Warning messages stripped for brevity.) >>> >>> [root@alpha1-p200 fio]# rbd -p ssd_replica rm image1 >>> >>> Removing image: 0% complete...failed. >>> >>> rbd: delete error: (2) No such file or directory >>> >>> [root@alpha1-p200 fio]# rados -p ssd_replica rm rbd_id.image1 >>> >>> error removing ssd_replica>rbd_id.image1: (2) No such file or directory >>> >>> [root@alpha1-p200 fio]# rados -p ssd_replica rm >>> rbd_data.15ed238e1f29.0000000000005b56 >>> >>> error removing ssd_replica>rbd_data.15ed238e1f29.0000000000005b56: (2) >>>No >>> such file or directory >>> >>> >>> Is this a bug, and if so, where should I look for the cause? Anyone >>>know >>> how to delete these objects? >>> >>> Thanks, Kevan >>> >>> >>> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> >> >>-- >>Jason > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |