Hello Zack, On Wed, Mar 6, 2019 at 1:18 PM Zack Brenton <zack@xxxxxxxxxxxx> wrote: > > Hello, > > We're running Ceph on Kubernetes 1.12 using the Rook operator (https://rook.io), but we've been struggling to scale applications mounting CephFS volumes above 600 pods / 300 nodes. All our instances use the kernel client and run kernel `4.19.23-coreos-r1`. > > We've tried increasing the MDS memory limits, running multiple active MDS pods, and running different versions of Ceph (up to the latest Luminous and Mimic releases), but we run into MDS_SLOW_REQUEST errors at the same scale regardless of the memory limits we set. See this GitHub issue for more info on what we've tried up to this point: https://github.com/rook/rook/issues/2590 > > I've written a simple load test that reads all the files in a given directory on an interval. While running this test, I've noticed that the `mds_co.bytes` value (from `ceph daemon mds.myfs-a dump_mempools | jq -c '.mempool.by_pool.mds_co'`) increases each time files are read. Why is this number increasing after the first iteration? If the same client is reading the same cached files, why would the data in the cache change at all? What is `mds_co.bytes` actually reporting? > > My most important question is this: How do I configure Ceph to be able to scale to large numbers of clients? Please post more information about your cluster: types of devices, `ceph osd tree`, `ceph osd df`, and `ceph osd lspools`. There's no reason why CephFS shouldn't be able to scale to that number of clients. The issue is probably related configuration of the pools/MDS. From your ticket, I have a *lot* of trouble believing the MDS still at 3GB memory usage with that number of clients and mds_cache_memory_limit=17179869184 (16GB). -- Patrick Donnelly _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |