On 09/12/2014 07:38 PM, Gregory Farnum wrote: > On Fri, Sep 12, 2014 at 6:49 AM, Florent Bautista <florent at coppint.com> wrote: >> Hi all, >> >> Today I have a problem using CephFS. I use firefly last release, with >> kernel 3.16 client (Debian experimental). >> >> I have a directory in CephFS, associated to a pool "pool2" (with >> set_layout). >> >> All is working fine, I can add and remove files, objects are stored in >> the right pool. >> >> But when Ceph cluster is overloaded (or for another reason, I don't >> know), sometimes when I remove a file, objects are not deleted in rados ! > CephFS file removal is asynchronous with you removing it from the > filesystem. The files get moved into a "stray" directory and will get > deleted once nobody holds references to them any more. My client is the only mounted and does not use files. This problems occurs when I delete files with "rm", but not when I use given rsync command. > >> I explain : I want to remove a large directory, containing millions of >> files. For a moment, objects are really deleted in rados (I see it in >> "rados df"), but when I start to do some heavy operations (like moving >> volumes in rdb), objects are not deleted anymore, "rados df" returns a >> fixed number of objects. I can see that files are still deleting because >> I use rsync (rsync -avP --stats --delete /empty/dir/ /dir/to/delete/). > What do you mean you're rsyncing and can see files deleting? I don't understand. When you run command I gave, syncing an empty dir with the dir you want deleted, rsync is telling you "Deleting (file)" for each file to unlink. > > Anyway, It's *possible* that the client is holding capabilities on the > deleted files and isn't handing them back, in which case unmounting it > would drop them (and then you could remount). I don't think we have > any commands designed to hasten that, though. unmounting does not help. When I unlink() via rsync, objects are deleted in rados (it makes all cluster slow down, and have slow requests). When I use rm command, it is much faster but objects are not deleted in rados ! When I re-mount root CephFS, there are no files, all empty. But still have 125 MB of objects in "metadata" pool and 21.57 GB in my data pool (and it does not decrease...)... I don't understand... > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com
 |