On Thu, Sep 18, 2014 at 10:39 AM, Florent B <florent at coppint.com> wrote: > On 09/12/2014 07:38 PM, Gregory Farnum wrote: >> On Fri, Sep 12, 2014 at 6:49 AM, Florent Bautista <florent at coppint.com> wrote: >>> Hi all, >>> >>> Today I have a problem using CephFS. I use firefly last release, with >>> kernel 3.16 client (Debian experimental). >>> >>> I have a directory in CephFS, associated to a pool "pool2" (with >>> set_layout). >>> >>> All is working fine, I can add and remove files, objects are stored in >>> the right pool. >>> >>> But when Ceph cluster is overloaded (or for another reason, I don't >>> know), sometimes when I remove a file, objects are not deleted in rados ! >> CephFS file removal is asynchronous with you removing it from the >> filesystem. The files get moved into a "stray" directory and will get >> deleted once nobody holds references to them any more. > > My client is the only mounted and does not use files. "does not use files"...what? > > This problems occurs when I delete files with "rm", but not when I use > given rsync command. > >> >>> I explain : I want to remove a large directory, containing millions of >>> files. For a moment, objects are really deleted in rados (I see it in >>> "rados df"), but when I start to do some heavy operations (like moving >>> volumes in rdb), objects are not deleted anymore, "rados df" returns a >>> fixed number of objects. I can see that files are still deleting because >>> I use rsync (rsync -avP --stats --delete /empty/dir/ /dir/to/delete/). >> What do you mean you're rsyncing and can see files deleting? I don't understand. > > When you run command I gave, syncing an empty dir with the dir you want > deleted, rsync is telling you "Deleting (file)" for each file to unlink. > >> >> Anyway, It's *possible* that the client is holding capabilities on the >> deleted files and isn't handing them back, in which case unmounting it >> would drop them (and then you could remount). I don't think we have >> any commands designed to hasten that, though. > > unmounting does not help. > > When I unlink() via rsync, objects are deleted in rados (it makes all > cluster slow down, and have slow requests). > > When I use rm command, it is much faster but objects are not deleted in > rados ! I think you're not doing what you think you're doing, then...those two actions should look the same to CephFS. > > When I re-mount root CephFS, there are no files, all empty. > > But still have 125 MB of objects in "metadata" pool and 21.57 GB in my > data pool (and it does not decrease...)... Well, the metadata pool is never going to be emptied; that holds your MDS journals. The data pool might not get entirely empty either; how many objects does it say it has?
 |