Dear Cephers, I have a Ceph cluster with 16 nodes, 335 OSDs all running Octopus 15.2.13 now. During upgrading from Nautilus last week, a problem was triggered by a specific node (Ceph09) and several slow requests were seen after upgrading Ceph09. The first 8 nodes have identical hardware and completed the upgrade process without problem. After the problem, client IO nearly stopped and rebooting this Ceph09 node did not help to solve the problem. The only thing that helped was rebooting monitor nodes one by one to get rid of these slow requests. We have seen lots of "fault initiating reconnect" messages like below from OSD logs of Ceph09 node. 2021-07-10T13:11:12.333+0300 7f90a5168700 0 --1- [v2: 10.30.3.139:6800/90655,v1:10.30.3.139:6801/90655] >> v1: 10.30.3.132:6833/286861 conn(0x561748e62c00 0x561768cc6800 :-1 s=OPENED pgs=2366 cs=245 l=0).fault initiating reconnect We have completed the upgrade process for other Ceph nodes without problem and all nodes are running Octopus 15.2.13 now. But when we restart OSDs of Ceph09 or reboot the node, the same problem occurs immediately. Any operation on the remaining nodes including rebooting node, restarting OSDS does not trigger the problem. Interestingly, we started to see the "fault initiating reconnect" messages for other OSDs running on different nodes after the problem. To investigate the problem, we tried to reweight all OSDs on Ceph09 to 0, the same problem has occurred again and we had slow requests and performance problems on client IO. To fix the problem, restarting all ceph daemons on monitors did not help. We rebooted monitor nodes one by one several times without luck. Finally, rebooting cephmonitor01 twice fixed the issue. We have checked all the network settings including MTUs and everything seems fine. iperf3 tests between any nodes in the cluster provides expected results. dmesg and syslog messages do not include any critical message about disks. At the time being, any operation on Ceph09 triggers the problem and we could not find a solution to fix the problem. Does anyone have any idea about this problem or any advice to trace the problem? Any advice and suggestions would be greatly appreciated. Best regards, HC _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |