|
Hi John, Some other symptoms of the problem: when the MDS has been running for a few days, it starts looking really busy. At this time, listing directories becomes really slow. An "ls -l" on a directory with about 250 entries takes about 2.5 seconds. All the metadata is on OSDs with NVMe backing stores. Interestingly enough the memory usage seems pretty low (compared to the allowed cache limit). PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1604408 ceph 20 0 3710304 2.387g 18360 S 100.0 0.9 757:06.92 /usr/bin/ceph-mds -f --cluster ceph --id cephmon00 --setuser ceph --setgroup ceph Once I bounce it (fail it over), the CPU usage goes down to the 10-25% range. The same ls -l after the bounce takes about 0.5 seconds. I remounted the filesystem before each test to ensure there isn't anything cached. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 111100 ceph 20 0 6537052 5.864g 18500 S 17.6 2.3 9:23.55 /usr/bin/ceph-mds -f --cluster ceph --id cephmon02 --setuser ceph --setgroup ceph Also, I have a crawler that crawls the file system periodically. Normally the full crawl runs for about 24 hours, but with the slowing down MDS, now it has been running for more than 2 days and isn't close to finishing. The MDS related settings we are running with are: mds_cache_memory_limit = 17179869184 Andras On 01/17/2018 01:11 PM, John Spray
wrote:
On Wed, Jan 17, 2018 at 3:36 PM, Andras Pataki <apataki@xxxxxxxxxxxxxxxxxxxxx> wrote:Hi John, All our hosts are CentOS 7 hosts, the majority are 7.4 with kernel 3.10.0-693.5.2.el7.x86_64, with fuse 2.9.2-8.el7. We have some hosts that have slight variations in kernel versions, the oldest one are a handful of CentOS 7.3 hosts with kernel 3.10.0-514.21.1.el7.x86_64 and fuse 2.9.2-7.el7. I know Redhat has been backporting lots of stuff so perhaps these kernels fall into the category you are describing?Quite possibly -- this issue was originally noticed on RHEL, so maybe the relevant bits made it back to CentOS recently. However, it looks like the fixes for that issue[1,2] are already in 12.2.2, so maybe this is something completely unrelated :-/ The ceph-fuse executable does create an admin command socket in /var/run/ceph (named something ceph-client...) that you can drive with "ceph daemon <socket> dump_cache", but the output is extremely verbose and low level and may not be informative. John 1. http://tracker.ceph.com/issues/21423 2. http://tracker.ceph.com/issues/22269When the cache pressure problem happens, is there a way to know exactly which hosts are involved, and what items are in their caches easily? Andras On 01/17/2018 06:09 AM, John Spray wrote:On Tue, Jan 16, 2018 at 8:50 PM, Andras Pataki <apataki@xxxxxxxxxxxxxxxxxxxxx> wrote:Dear Cephers, We've upgraded the back end of our cluster from Jewel (10.2.10) to Luminous (12.2.2). The upgrade went smoothly for the most part, except we seem to be hitting an issue with cephfs. After about a day or two of use, the MDS start complaining about clients failing to respond to cache pressure:What's the OS, kernel version and fuse version on the hosts where the clients are running? There have been some issues with ceph-fuse losing the ability to properly invalidate cached items when certain updated OS packages were installed. Specifically, ceph-fuse checks the kernel version against 3.18.0 to decide which invalidation method to use, and if your OS has backported new behaviour to a low-version-numbered kernel, that can confuse it. John |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
