On Tue, Dec 13, 2016 at 6:03 AM, Goncalo Borges
<goncalo.borges@xxxxxxxxxxxxx> wrote:
> Hi Ceph(FS)ers...
>
> I am currently running in production the following environment:
>
> - ceph/cephfs in 10.2.2.
> - All infrastructure is in the same version (rados cluster, mons, mds and
> cephfs clients).
> - We mount cephfs using ceph-fuse.
>
> Since yesterday that we have our cluster in warning state with the message
> "mds0: Many clients (X) failing to respond to cache pressure". X has been
> changing with time, from ~130 to ~70. I am able to correlate the appearance
> of this message with burst of jobs in our cluster.
>
> This subject has been discussed in the mailing list a lot of times, and
> normally, the recipe is to look for something wrong in the clients. So, I
> have tried to look to clients first:
>
> 1) I've started to loop through all my clients, and run 'ceph --admin-daemon
> /var/run/ceph/ceph-client.mount_user.asok status' to get the inodes_count
> reported in each client.
>
> $ cat all.txt | grep inode_count | awk '{print $2}' | sed 's/,//g' | awk
> '{s+=$1} END {print s}'
> 2407659
>
> 2) I've then compared with the number of inodes the mds had in its cache
> (obtained by a perf dump)
> inode_max": 2000000 and "inodes": 2413826
>
> 3) I've tried to understand how many clients had a number of inodes higher
> than 16384 (the default) and got
>
> $ for i in `cat all.txt | grep inode_count | awk '{print $2}' | sed 's/,//g'
> `; do if [ $i -ge 16384 ]; then echo $i; fi; done | wc -l
> 27
>
> 4) My conclusion is that the core of inodes is held by a couple of machines.
> However, while the majority is running user jobs, others are not doing
> anything at all. For example, an idle machine (which had no users logged in,
> no jobs running, updatedb does not search for cephfs filesystem) reported
> more than > 300000 inodes). To regain those inodes, I had to umount and
> remount cephfs in that machine.
>
> 5) Based on my previous observations I suspect that there are still some
> problems in the ceph-fuse client regarding recovering these inodes (or it
> happens at a very slow rate).
Seems that way. Can you come up with a reproducer for us, and/or
gather some client+mds debug logs where a client is failing to respond
to cache pressure?
Also, what kernel is in use on the clients? It's possible that the
issue is in FUSE itself (or the way that it responses to ceph-fuse's
attempts to ask for some inodes to be released).
> However, I also do not completely understand what is happening on the server
> side:
>
> 6) The current memory usage of my mds is the following:
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 17831 ceph 20 0 13.667g 0.012t 10048 S 37.5 40.2 1068:47 ceph-mds
>
> The mds cache size is set to 2000000. Running 'ceph daemon mds.<id> perf
> dump', I get "inode_max": 2000000 and "inodes": 2413826. Assuming 4k per
> each inode one gets ~10G. So why it is taking much more than that?
>
>
> 7) I have been running cephfs for more than an year, and looking to ganglia,
> the mds memory never decreases but always increases (even in cases when we
> umount almost all the clients). Why does that happen?
Coincidentally someone posted about this on ceph-devel just yesterday.
The answer is that the MDS uses memory pools for allocation, and it
doesn't (currently) ever bother releasing memory back to the operating
system because it's doing its own cache size enforcement. However,
when the cache size limits aren't being enforced (for example because
of clients failing to release caps) this becomes a problem. There's a
patch for master (https://github.com/ceph/ceph/pull/12443)
>
> 8) I am running 2 mds, in active / standby-replay mode. The memory of the
> standby-replay is much lower
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 716 ceph 20 0 6149424 5.115g 8524 S 1.2 43.6 53:19.74 ceph-mds
>
> If I trigger a restart on my active mds, the standby replay will start
> acting as active, but will continue with the same amount of memory. Why the
> second mds can become active, and do the same job but using much more
> memory?
Presumably this also makes sense once you know about the allocator in use.
> 9) Finally, I am sending an extract of 'ceph daemon mds.<id> perf dump' from
> my active and standby mdses. What is exactly the meaning of inodes_pin_tail,
> inodes_expired and inodes_with_caps? Is the standby mds suppose to show the
> same numbers? They don't...
It's not really possible to explain these counters without a
substantial explanation of MDS internals, sorry. I will say though
that there is absolutely no guarantee of performance counters on the
standby replay daemon matching those on the active daemon.
John
> Thanks in advance for your answers / suggestions
>
> Cheers
>
> Goncalo
>
>
>
> active:
>
> "mds": {
> "request": 93941296,
> "reply": 93940671,
> "reply_latency": {
> "avgcount": 93940671,
> "sum": 188398.004552299
> },
> "forward": 0,
> "dir_fetch": 309878,
> "dir_commit": 1736194,
> "dir_split": 0,
> "inode_max": 2000000,
> "inodes": 2413826,
> "inodes_top": 201,
> "inodes_bottom": 568,
> "inodes_pin_tail": 2413057,
> "inodes_pinned": 2413303,
> "inodes_expired": 19693168,
> "inodes_with_caps": 2409737,
> "caps": 2440565,
> "subtrees": 2,
> "traverse": 113291068,
> "traverse_hit": 57822611,
> "traverse_forward": 0,
> "traverse_discover": 0,
> "traverse_dir_fetch": 154708,
> "traverse_remote_ino": 1085,
> "traverse_lock": 66063,
> "load_cent": 9394314733,
> "q": 22,
> "exported": 0,
> "exported_inodes": 0,
> "imported": 0,
> "imported_inodes": 0
> },
>
> standby-replay:
>
> "mds": {
> "request": 0,
> "reply": 0,
> "reply_latency": {
> "avgcount": 0,
> "sum": 0.000000000
> },
> "forward": 0,
> "dir_fetch": 0,
> "dir_commit": 0,
> "dir_split": 0,
> "inode_max": 2000000,
> "inodes": 2000058,
> "inodes_top": 0,
> "inodes_bottom": 1993207,
> "inodes_pin_tail": 6851,
> "inodes_pinned": 124135,
> "inodes_expired": 10651484,
> "inodes_with_caps": 0,
> "caps": 0,
> "subtrees": 2,
> "traverse": 0,
> "traverse_hit": 0,
> "traverse_forward": 0,
> "traverse_discover": 0,
> "traverse_dir_fetch": 0,
> "traverse_remote_ino": 0,
> "traverse_lock": 0,
> "load_cent": 0,
> "q": 0,
> "exported": 0,
> "exported_inodes": 0,
> "imported": 0,
> "imported_inodes": 0
> },
>
>
>
>
>
> --
> Goncalo Borges
> Research Computing
> ARC Centre of Excellence for Particle Physics at the Terascale
> School of Physics A28 | University of Sydney, NSW 2006
> T: +61 2 93511937
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
