Thanks!
Now this slow rm is not our biggest issue anymore (for the moment)..
Since this night all our MDSs did crash.
I opened a ticket http://tracker.ceph.com/issues/15379 with the
stacktraces I got.
We aren't able to restart the mds's at all for now..
I stopped the rm, and also increased the mds_beacon_grace because it was
first complaining about timeouts.
Without luck, still not able to keep the mds's up longer than 5minutes..
How can we solve this ?
Thanks again!
Kenneth
On 05/04/16 04:19, Yan, Zheng wrote:
On Tue, Apr 5, 2016 at 12:55 AM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
Deletes are just slow right now. You can look at the ops in flight on you
client or MDS admin socket to see how far along it is and watch them to see
how long stuff is taking -- I think it's a sync disk commit for each unlink
though so at 40M it's going to be a good looong while. :/
-Greg
On Monday, April 4, 2016, Kenneth Waegeman <kenneth.waegeman@xxxxxxxx>
wrote:
Hi all,
I want to remove a large directory containing +- 40M files /160TB of data
in CephFS by running rm -rf on the directory via the ceph kernel client.
After 7h , the rm command is still running. I checked the rados df output,
and saw that only about 2TB and 2M files are gone.
I know this output of rados df can be confusing because ceph should delete
objects asyncroniously, but then I don't know why the rm command still
hangs.
Is there some way to speed this up? And is there a way to check how far
the marked for deletion has progressed ?
Check /sys/kernel/debug/ceph/xxx/mdsc, you can roughly get how fast
unlink requests are handled. If MDS's CPU usage is less than 100%.
you can try running multiple instance of 'rm -rf' (each one removes
different sub-directory)
Regards
Yan, Zheng
Thank you very much!
Kenneth
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
