Re: CephFS: No space left on device

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Sun, Oct 2, 2016 at 11:09 AM, Mykola Dvornik
<mykola.dvornik@xxxxxxxxx> wrote:
> After upgrading to 10.2.3 we frequently see messages like

>From which version did you upgrade?

> 'rm: cannot remove '...': No space left on device
>
> The folders we are trying to delete contain approx. 50K files 193 KB each.

My guess would be that you are hitting the new
mds_bal_fragment_size_max check.  This limits the number of entries
that the MDS will create in a single directory fragment, to avoid
overwhelming the OSD with oversized objects.  It is 100000 by default.
This limit also applies to "stray" directories where unlinked files
are put while they wait to be purged, so you could get into this state
while doing lots of deletions.  There are ten stray directories that
get a roughly even share of files, so if you have more than about one
million files waiting to be purged, you could see this condition.

The "Client failing to respond to cache pressure" messages may play a
part here -- if you have misbehaving clients then they may cause the
MDS to delay purging stray files, leading to a backlog.  If your
clients are by any chance older kernel clients, you should upgrade
them.  You can also unmount/remount them to clear this state, although
it will reoccur until the clients are updated (or until the bug is
fixed, if you're running latest clients already).

The high level counters for strays are part of the default output of
"ceph daemonperf mds.<id>" when run on the MDS server (the "stry" and
"purg" columns).  You can look at these to watch how fast the MDS is
clearing out strays.  If your backlog is just because it's not doing
it fast enough, then you can look at tuning mds_max_purge_files and
mds_max_purge_ops to adjust the throttles on purging.  Those settings
can be adjusted without restarting the MDS using the "injectargs"
command (http://docs.ceph.com/docs/master/rados/operations/control/#mds-subsystem)

Let us know how you get on.

John


> The cluster state and storage available are both OK:
>
>     cluster 98d72518-6619-4b5c-b148-9a781ef13bcb
>      health HEALTH_WARN
>             mds0: Client XXX.XXX.XXX.XXX failing to respond to cache
> pressure
>             mds0: Client XXX.XXX.XXX.XXX failing to respond to cache
> pressure
>             mds0: Client XXX.XXX.XXX.XXX failing to respond to cache
> pressure
>             mds0: Client XXX.XXX.XXX.XXX failing to respond to cache
> pressure
>             mds0: Client XXX.XXX.XXX.XXX failing to respond to cache
> pressure
>      monmap e1: 1 mons at {000-s-ragnarok=XXX.XXX.XXX.XXX:6789/0}
>             election epoch 11, quorum 0 000-s-ragnarok
>       fsmap e62643: 1/1/1 up {0=000-s-ragnarok=up:active}
>      osdmap e20203: 16 osds: 16 up, 16 in
>             flags sortbitwise
>       pgmap v15284654: 1088 pgs, 2 pools, 11263 GB data, 40801 kobjects
>             23048 GB used, 6745 GB / 29793 GB avail
>                 1085 active+clean
>                    2 active+clean+scrubbing
>                    1 active+clean+scrubbing+deep
>
>
> Has anybody experienced this issue so far?
>
> Regards,
> --
>  Mykola
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux