About how many files are we talking here?
Implementation detail on file deletion to understand why this might happen: deletion is async, deleting a file inserts it into the purge queue and the actual data will be removed in the background.
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Fri, May 10, 2019 at 1:04 PM Kenneth Waegeman <kenneth.waegeman@xxxxxxxx> wrote:
Hi all,
I am seeing issues on cephfs running 13.2.5 when deleting files:
[root@osd006 ~]# rm /mnt/ceph/backups/osd006.gigalith.os-2b5a3740.1326700
rm: remove regular empty file
‘/mnt/ceph/backups/osd006.gigalith.os-2b5a3740.1326700’? y
rm: cannot remove
‘/mnt/ceph/backups/osd006.gigalith.os-2b5a3740.1326700’: No space left
on device
few minutes later, I can remove it without problem. This happens
especially when there are a lot of files deleted somewhere on the
filesystem around the same time.
We already have tuned our mds config:
[mds]
mds_cache_memory_limit=10737418240
mds_log_max_expiring=200
mds_log_max_segments=200
mds_max_purge_files=2560
mds_max_purge_ops=327600
mds_max_purge_ops_per_pg=20
ceph -s is reporting everything clean, and the file system space usage
is less than 50%, also no full osds or anything.
Is there a way to further debug what the bottleneck is when removing
files that gives this 'no space left on device' error?
Thank you very much!
Kenneth
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
