Today we pulled the wrong disk from a ceph node. And that made the whole node go down/be unresponsive. Even to a simple ping. I cannot find to much about this in the log files. But I expect that the /usr/bin/ceph-osd process caused a kernel panic. Linux c01 3.10.0-693.11.1.el7.x86_64 CentOS Linux release 7.4.1708 (Core) libcephfs2-12.2.7-0.el7.x86_64 ceph-mon-12.2.7-0.el7.x86_64 nfs-ganesha-ceph-2.6.1-0.1.el7.x86_64 ceph-selinux-12.2.7-0.el7.x86_64 ceph-osd-12.2.7-0.el7.x86_64 ceph-mgr-12.2.7-0.el7.x86_64 ceph-12.2.7-0.el7.x86_64 python-cephfs-12.2.7-0.el7.x86_64 ceph-common-12.2.7-0.el7.x86_64 ceph-mds-12.2.7-0.el7.x86_64 ceph-radosgw-12.2.7-0.el7.x86_64 ceph-base-12.2.7-0.el7.x86_64 Aug 1 11:01:01 c02 systemd: Started Session 8331 of user root. Aug 1 11:01:01 c02 systemd: Starting Session 8331 of user root. Aug 1 11:01:01 c02 systemd: Starting Session 8331 of user root. Aug 1 11:03:08 c03 kernel: XFS (sdb1): xfs_do_force_shutdown(0x2) called from line 1200 of file fs/xfs/xfs_log.c. Return address = 0xffffffffc0232e60 Aug 1 11:03:08 c03 kernel: XFS (sdb1): xfs_do_force_shutdown(0x2) called from line 1200 of file fs/xfs/xfs_log.c. Return address = 0xffffffffc0232e60 Aug 1 11:03:33 c03 kernel: XFS (sdf1): xfs_do_force_shutdown(0x2) called from line 1200 of file fs/xfs/xfs_log.c. Return address = 0xffffffffc0232e60 Aug 1 11:03:33 c03 kernel: XFS (sdf1): xfs_do_force_shutdown(0x2) called from line 1200 of file fs/xfs/xfs_log.c. Return address = 0xffffffffc0232e60 Aug 1 11:03:34 c02 kernel: libceph: osd5 down Aug 1 11:03:34 c02 kernel: libceph: osd5 down Aug 1 11:05:04 c02 ceph-osd: 2018-08-01 11:05:04.656719 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6816 osd.12 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:44.656717) Aug 1 11:05:04 c02 ceph-osd: 2018-08-01 11:05:04.656719 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6816 osd.12 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:44.656717) Aug 1 11:05:04 c02 ceph-osd: 2018-08-01 11:05:04.656746 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6812 osd.14 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:44.656717) Aug 1 11:05:04 c02 ceph-osd: 2018-08-01 11:05:04.656761 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6804 osd.15 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:44.656717) Aug 1 11:05:04 c02 ceph-osd: 2018-08-01 11:05:04.656773 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6814 osd.16 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:44.656717) Aug 1 11:05:04 c02 ceph-osd: 2018-08-01 11:05:04.656746 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6812 osd.14 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:44.656717) Aug 1 11:05:04 c02 ceph-osd: 2018-08-01 11:05:04.656761 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6804 osd.15 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:44.656717) Aug 1 11:05:04 c02 ceph-osd: 2018-08-01 11:05:04.656773 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6814 osd.16 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:44.656717) Aug 1 11:05:05 c02 ceph-osd: 2018-08-01 11:05:05.657034 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6816 osd.12 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:45.657031) Aug 1 11:05:05 c02 ceph-osd: 2018-08-01 11:05:05.657034 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6816 osd.12 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:45.657031) Aug 1 11:05:05 c02 ceph-osd: 2018-08-01 11:05:05.657067 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6812 osd.14 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:45.657031) Aug 1 11:05:05 c02 ceph-osd: 2018-08-01 11:05:05.657079 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6804 osd.15 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:45.657031) Aug 1 11:05:05 c02 ceph-osd: 2018-08-01 11:05:05.657089 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6814 osd.16 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:45.657031) Aug 1 11:05:05 c02 ceph-osd: 2018-08-01 11:05:05.657067 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6812 osd.14 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:45.657031) Aug 1 11:05:05 c02 ceph-osd: 2018-08-01 11:05:05.657079 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6804 osd.15 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:45.657031) Aug 1 11:05:05 c02 ceph-osd: 2018-08-01 11:05:05.657089 7f1f1e764700 -1 osd.9 22452 heartbeat_check: no reply from 192.168.10.113:6814 osd.16 since back 2018-08-01 11:04:44.365869 front 2018-08-01 11:04:44.365869 (cutoff 2018-08-01 11:04:45.657031) Aug 1 11:05:06 c02 ceph-osd: 2018-08-01 11:05:06.420853 7f9e2612b700 -1 osd.20 22452 heartbeat_check: no reply from 192.168.10.113:6810 osd.13 since back 2018-08-01 11:04:45.894147 front 2018-08-01 11:04:45.894147 (cutoff 2018-08-01 11:04:46.420850) _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |