Dear Cephers, I am running IO500 testing on CephFS 3-node cluster with version Nautilus 14.2.22, and during the tests, we observed a lot OSD down and pgremapping, and looking at OSD logs, we see the followings: How can we avoid such instability or issues? any suggestions or comments are highly appreciated. Samuel [104592.457005] libceph: osd10 192.168.10.35:6820 socket closed (con state OPEN) [104592.628277] libceph: osd10 192.168.10.35:6820 socket closed (con state CONNECTING) [104593.453087] libceph: osd10 192.168.10.35:6820 socket closed (con state CONNECTING) [104593.488436] libceph: osd8 192.168.10.35:6843 socket closed (con state OPEN) [104593.803299] libceph: osd8 192.168.10.35:6843 socket closed (con state CONNECTING) [104594.413115] libceph: osd8 192.168.10.35:6843 socket closed (con state CONNECTING) [104594.478092] libceph: osd10 192.168.10.35:6820 socket closed (con state CONNECTING) [104595.437164] libceph: osd8 192.168.10.35:6843 socket closed (con state CONNECTING) [104596.461221] libceph: osd10 192.168.10.35:6820 socket closed (con state CONNECTING) [104597.421188] libceph: osd8 192.168.10.35:6843 socket closed (con state CONNECTING) [104600.429246] libceph: osd10 192.168.10.35:6820 socket closed (con state CONNECTING) [104601.453162] libceph: osd8 192.168.10.35:6843 socket closed (con state CONNECTING) [104607.556989] libceph: osd10 down [104608.500131] libceph: osd11 192.168.10.35:6851 socket closed (con state OPEN) [104608.892067] libceph: osd11 192.168.10.35:6851 socket closed (con state CONNECTING) [104609.309276] libceph: osd11 192.168.10.35:6851 socket closed (con state CONNECTING) [104610.349239] libceph: osd11 192.168.10.35:6851 socket closed (con state CONNECTING) [104612.334302] libceph: osd11 192.168.10.35:6851 socket closed (con state CONNECTING) [104613.490471] libceph: osd3 192.168.10.34:6821 socket closed (con state OPEN) [104613.661655] libceph: osd3 192.168.10.34:6821 socket closed (con state CONNECTING) [104614.381240] libceph: osd3 192.168.10.34:6821 socket error on write [104615.342232] libceph: osd3 192.168.10.34:6821 socket error on write [104616.365433] libceph: osd11 192.168.10.35:6851 socket closed (con state CONNECTING) [104617.389321] libceph: osd3 192.168.10.34:6821 socket error on write [104621.357310] libceph: osd3 192.168.10.34:6821 socket error on write [104624.957004] libceph: osd0 192.168.10.34:6852 socket closed (con state OPEN) [104625.119796] libceph: osd0 192.168.10.34:6852 socket closed (con state CONNECTING) [104626.349357] libceph: osd0 192.168.10.34:6852 socket error on write [104627.373508] libceph: osd0 192.168.10.34:6852 socket error on write [104629.357432] libceph: osd0 192.168.10.34:6852 socket error on write [104629.421412] libceph: osd3 192.168.10.34:6821 socket error on write [104631.458368] libceph: osd8 down [104631.458370] libceph: osd11 down [104631.680203] libceph: osd13 down [104633.389445] libceph: osd0 192.168.10.34:6852 socket error on write [104633.391371] libceph: osd0 down [104633.391372] libceph: osd3 down [104636.851429] libceph: osd8 up [104636.851431] libceph: osd11 up [104636.906442] libceph: osd10 up huxiaoyu@xxxxxxxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |