Re: Force cephfs delayed deletion

Alexander Ryabov <aryabov@xxxxxxxxxxxxxx> · Thu, 19 Jul 2018 15:08:44 +0000

>Also, since I see this is a log directory, check that you don't have some processes that are holding their log
 files open even after they're unlinked.

Thank you very much - that was the case.

lsof /mnt/logs | grep deleted

After dealing with these, space was reclaimed in about 2-3min.

From: John Spray <jspray@xxxxxxxxxx>

Sent: Thursday, July 19, 2018 17:24

To: Alexander Ryabov

Cc: ceph-users@xxxxxxxxxxxxxx

Subject: Re:  Force cephfs delayed deletion

On Thu, Jul 19, 2018 at 1:58 PM Alexander Ryabov <aryabov@xxxxxxxxxxxxxx> wrote:

Hello,

I see that free space is not released after files are removed on CephFS.

I'm using Luminous with replica=3 without any snapshots etc and with default settings.

From client side:

$ du -sh /mnt/logs/
4.1G
/mnt/logs/

$ df -h /mnt/logs/
Filesystem   Size  Used Avail Use% Mounted on
h1,h2:/logs  125G   87G   39G  70% /mnt/logs

These stats are after couple of large files were removed in /mnt/logs dir, but that only dropped Useв space a little.

Check what version of the client you're using -- some older clients had bugs that would hold references to deleted files and prevent them from being purged.  If you find that the space starts getting freed when you unmount the client, this is likely to
 be because of a client bug.

Also, since I see this is a log directory, check that you don't have some processes that are holding their log files open even after they're unlinked.

John

Doing 'sync' command also changes nothing.

From server side:

# ceph  df
GLOBAL:
    SIZE     AVAIL      RAW USED     %RAW USED
    124G     39226M       88723M         69.34
POOLS:
    NAME                ID     USED       %USED     MAX AVAIL     OBJECTS
    cephfs_data         1      28804M     76.80         8703M        7256
    cephfs_metadata     2        236M      2.65         8703M         101

Why there are such a large difference between 'du' and 'USED'?

I've found that it could be due to 'delayed delete' http://docs.ceph.com/docs/luminous/dev/delayed-delete/
And previously it seems could be tuned by adjusting the "mds max purge files" and "mds max purge ops" 

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-October/013679.html

But there is no more of such options in http://docs.ceph.com/docs/luminous/cephfs/mds-config-ref/

So the question is - how to purge deleted data and reclaim free space?
Thank you.

______________________________________________
ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Intermedia

10th floor, 2, Alexander Nevsky Sq.

Saint-Petersburg, Russia 191167

www.intermedia.net

J.D. Power certifies Intermedia for technical support excellence two years running, a first among cloud application providers

This message is intended only for the person(s) to which it is addressed and may contain Intermedia.net. Inc. privileged, confidential and/or proprietary information. If you have received this communication in error, please notify us immediately by replying
 to the message and deleting it from your computer. Any disclosure, copying, distribution, or the taking of any action concerning the contents of this message and any attachment(s) by anyone other than the named recipient(s) is strictly prohibited.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com