Sorry, there is also a graph of the metadata that wasn't uploaded by mistake. On Mon, 18 Feb 2019 at 16:08, Theofilos Mouratidis <mtheofilos@xxxxxxxxx> wrote: > > Hello Cephers, > > Here at CERN we enabled the MDS balancing after v12.2.8, which > included some improvements to the balancing code. In our cephfs setup > we have 6 mds machines with multi-mds enabled (5 active + 1 standby). > (Prior to enabling mds balancing we used pinning to pin manila > directories to a random mds). Since we disabled the pinning the > performance has generally degraded, and we think we know why: > > 1. When a directory is exported from mds a to b, this drops the inode > caches for the directory from mds a. > 2. Because of this, the MDS's need significantly more metadata IO to > keep loading inodes from RADOS. (metadata is on hdd's in this cluster) > 3. Our workloads are quite dynamic and the balancer never manages to > find a stable set of exports; it is constantly churning the exports > between MDS's. > > I have attached 2 images that show the mds ram usage and the metadata > io. You can see that before, with mds pinning, the RAM usage is stable > (so the inode caches are effective). After disabling pinning, the RAM > usage is fluctuating and metadata IO is much more intensive than > before. > > We think this comes from the general goal of the MDS balancing -- it > tries to balance requests evenly across MDSs but has no concern for > the inode caches in the MDS's. Do you additional metrics such as cache > hit rate or memory usage should be added to the load metric? Or rather > is this something that can be fixed with some mds_bal_* parameter fine > tuning? > > Thanks, > Theofilos Mouratidis > Software Engineer, Storage Group at CERN
Attachment:
mds io.png
Description: PNG image
