Can you grab the perf dump during this time, perhaps plus dumps of the ops in progress?
This is weird but given it’s somewhat periodic it might be something like the MDS needing to catch up on log trimming (though I’m unclear why changing the cache size would impact this).
On Sun, Sep 30, 2018 at 9:02 PM Adam Tygart <mozes@xxxxxxx> wrote:
Hello all,
I've got a ceph (12.2.8) cluster with 27 servers, 500 osds, and 1000
cephfs mounts (kernel client). We're currently only using 1 active
mds.
Performance is great about 80% of the time. MDS responses (per ceph
daemonperf mds.$(hostname -s), indicates 2k-9k requests per second,
with a latency under 100.
It is the other 20ish percent I'm worried about. I'll check on it and
it with be going 5-15 seconds with "0" requests, "0" latency, then
give me 2 seconds of reasonable response times, and then back to
nothing. Clients are actually seeing blocked requests for this period
of time.
The strange bit is that when I *reduce* the mds_cache_size, requests
and latencies go back to normal for a while. When it happens again,
I'll increase it back to where it was. It feels like the mds server
decides that some of these inodes can't be dropped from the cache
unless the cache size changes. Maybe something wrong with the LRU?
I feel like I've got a reasonable cache size for my workload, 30GB on
the small end, 55GB on the large. No real reason for a swing this
large except to potentially delay it recurring after expansion for
longer.
I also feel like there is probably some magic tunable to change how
inodes get stuck in the LRU. perhaps mds_cache_mid. Anyone know what
this tunable actually does? The documentation is a little sparse.
I can grab logs from the mds if needed, just let me know the settings
you'd like to see.
--
Adam
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
