On 24/05/2023 21.15, Emmanuel Jaep wrote: > Hi, > > we are currently running a ceph fs cluster at the following version: > MDS version: ceph version 16.2.10 > (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) > > The cluster is composed of 7 active MDSs and 1 standby MDS: > RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS > 0 active icadmin012 Reqs: 73 /s 1938k 1880k 85.3k 92.8k > 1 active icadmin008 Reqs: 206 /s 2375k 2375k 7081 171k > 2 active icadmin007 Reqs: 91 /s 5709k 5256k 149k 299k > 3 active icadmin014 Reqs: 93 /s 679k 664k 40.1k 216k > 4 active icadmin013 Reqs: 86 /s 3585k 3569k 12.7k 197k > 5 active icadmin011 Reqs: 72 /s 225k 221k 8611 164k > 6 active icadmin015 Reqs: 87 /s 1682k 1610k 27.9k 274k > POOL TYPE USED AVAIL > cephfs_metadata metadata 8552G 22.3T > cephfs_data data 226T 22.3T > STANDBY MDS > icadmin006 > > When I restart one of the active MDSs, the standby MDS becomes active and > its state becomes "replay". So far, so good! > > However, only one of the other "active" MDSs seems to remain active. All > activities drop from the other ones: > RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS > 0 active icadmin012 Reqs: 0 /s 1938k 1881k 85.3k 9720 > 1 active icadmin008 Reqs: 0 /s 2375k 2375k 7080 2505 > 2 active icadmin007 Reqs: 2 /s 5709k 5256k 149k 26.5k > 3 active icadmin014 Reqs: 0 /s 679k 664k 40.1k 3259 > 4 replay icadmin006 801k 801k 1279 0 > 5 active icadmin011 Reqs: 0 /s 225k 221k 8611 9241 > 6 active icadmin015 Reqs: 0 /s 1682k 1610k 27.9k 34.8k > POOL TYPE USED AVAIL > cephfs_metadata metadata 8539G 22.8T > cephfs_data data 225T 22.8T > STANDBY MDS > icadmin013 > > In effect, the cluster becomes almost unavailable until the newly promoted > MDS finishes rejoining the cluster. > > Obviously, this defeats the purpose of having 7MDSs. > Is this behavior? > If not, what configuration items should I check to go back to "normal" > operations? If *any* active MDS is down at least chunks of your filesystem will be down, which means clients will likely hang and stop doing anything even if the other MDSes are capable of serving subsets of the filesystem. Active MDSes do not randomly balance requests, they are each in charge of a subset of the filesystem and they all must be up for the filesystem to work. If you want reliability with fast failover, you need standby MDSes. - Hector _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |