On Fri, Oct 23, 2015 at 7:08 AM, Burkhard Linke <Burkhard.Linke@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote: > Hi, > > On 10/14/2015 06:32 AM, Gregory Farnum wrote: >> >> On Mon, Oct 12, 2015 at 12:50 AM, Burkhard Linke >> <Burkhard.Linke@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote: >>> >>> > *snipsnap* >>> >>> Thanks, that did the trick. I was able to locate the host blocking the >>> file >>> handles and remove the objects from the EC pool. >>> >>> Well, all except one: >>> >>> # ceph df >>> ... >>> ec_ssd_cache 18 4216k 0 2500G 129 >>> cephfs_ec_data 19 4096k 0 31574G 1 >>> >>> # rados -p ec_ssd_cache ls >>> 10000ef540f.00000386 >>> # rados -p cephfs_ec_data ls >>> 10000ef540f.00000386 >>> # ceph mds tell cb-dell-pe620r dumpcache cache.file >>> # grep 10000ef540f /cache.file >>> # >>> >>> It does not show up in the dumped cache file, but keeps being promoted to >>> the cache tier after MDS restarts. I've restarted most of the cephfs >>> clients >>> by unmounting cephfs and restarting ceph-fuse, but the object remains >>> active. >> >> You can enable MDS debug logging and see if the inode shows up in the >> log during replay. It's possible it's getting read in (from journal >> operations) but then getting evicted from cache if nobody's accessing >> it any more. >> You can also look at the xattrs on the object to see what the >> backtrace is and if that file is in cephfs. > > After the last MDS restart the stray object was not promoted to the cache > anymore: > ec_ssd_cache 18 120k 0 3842G 128 > cephfs_ec_data 19 4096k 0 10392G 1 > > There are no xattrs available for the stray object, so it's not possible to > find out which file it belongs/belonged to: > # rados -p cephfs_ec_data ls > 10000ef540f.00000386 > # rados -p cephfs_ec_data listxattr 10000ef540f.00000386 > # > > Is it possible to list pending journal operations to be on the safe side? Check out the cephfs-journal-tool. I don't remember the exact commands but I think it has good help text. -Greg _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |