I don't think # clients alone is a good measure by which to decide to deploy multiple MDSs -- idle clients create very little load, but just a few badly behaving clients can use all the MDS performance. (If you must hear a number, I can share that we have single MDSs with 2-3000 clients connected.) To detect an overloaded MDS, in most cases the users will notice that metadata ops are becoming slow -- so simple things like ls, mv, or creating files will become slow. The MDS cpu usage will also be very high -- (note that the MDS is not multithreaded, so at most it will saturate somewhere between 100-200%). There are also some op latency metrics in the mds perf dump you can observe -- if simple ops are taking more than a few milliseconds then this is another indication that the MDS load is too high. Next, if you do find that you need multiple MDSs, and if you understand the client workload very well, it is best if you can use subtree pinning to statically pin sub directories to a particular MDS rank. See https://ceph.io/community/new-luminous-cephfs-subtree-pinning/ Without pinning, the MDSs will use a heuristic to move subtrees between themselves -- this doesn't always work very well for all workloads, and can sometimes cause more harm than good. Cheers, Dan On Thu, May 27, 2021 at 9:30 AM Andres Rojas Guerrero <a.rojas@xxxxxxx> wrote: > > Oh, very interesting!! I have reduced the number of MDS to one. Only one > question more, out of curiosity, from what number can we consider that > there are many clients? > > > > El 27/5/21 a las 9:24, Dan van der Ster escribió: > > On Thu, May 27, 2021 at 9:21 AM Andres Rojas Guerrero <a.rojas@xxxxxxx> wrote: > >> > >> > >> > >> El 26/5/21 a las 16:51, Dan van der Ster escribió: > >>> I see you have two active MDSs. Is your cluster more stable if you use > >>> only one single active MDS? > >> > >> Good question!! I read form Ceph Doc: > >> > >> "You should configure multiple active MDS daemons when your metadata > >> performance is bottlenecked on the single MDS that runs by default." > >> > >> "Workloads that typically benefit from a larger number of active MDS > >> daemons are those with many clients, perhaps working on many separate > >> directories." > >> > >> I have more or less 25 concurrent clients, but working in the same > >> directory, Is that number a lot of clients? > >> > >> And I assumed that two are always better than one. > > > > 25 isn't many clients, but if they are operating in the same directory > > it will create a lot of contention between the two MDSs, which might > > explain some of the issues you observe. > > I recommend that you reduce back to 1 active mds and observe the > > system stability and performance. > > > > -- dan > > > > -- > ******************************************************* > Andrés Rojas Guerrero > Unidad Sistemas Linux > Area Arquitectura Tecnológica > Secretaría General Adjunta de Informática > Consejo Superior de Investigaciones Científicas (CSIC) > Pinar 19 > 28006 - Madrid > Tel: +34 915680059 -- Ext. 990059 > email: a.rojas@xxxxxxx > ID comunicate.csic.es: @50852720l:matrix.csic.es > ******************************************************* _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |