Hello. Half a hour ago, 7/28 of my servers are crashed (because of corosync! "2.4.4-3") and 2 of them was MON, I have 3 MON on my cluster. After they come back, I see high disk utilization because of ceph-osd processes. All of my cluster is not responding right now! All of my OSDs are consuming all the disk I/Os. I stopped all of the OSDs (168 of them). And then I set norecover and norebalance. After then started OSDs but nothing has changed. What should I do now? Here is my ceph.conf: https://paste.ubuntu.com/p/8RJ5c78rN5/ Here is the ceph -w: https://paste.ubuntu.com/p/NCKBbQgW3y/ Here is the ceph health detail: https://paste.ubuntu.com/p/BsyHNwVS26/ Here is the iostat -xd 1: https://paste.ubuntu.com/p/nrwd9bntyM/ https://pasteboard.co/HFvutu0.jpg _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |