Hello Now the OSD down again and show following log for the osd. Nov 12 20:51:34 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:51:34.009+0600 7f17dd634700 -1 osd.10 239591 set_numa_affinity unable to identify public interface '' n Nov 12 20:52:43 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:52:43.075+0600 7f17dd634700 -1 osd.10 239597 set_numa_affinity unable to identify public interface '' n Nov 12 20:53:53 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:53:53.066+0600 7f17dd634700 -1 osd.10 239603 set_numa_affinity unable to identify public interface '' n Nov 12 20:55:01 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:55:01.640+0600 7f17dd634700 -1 osd.10 239611 set_numa_affinity unable to identify public interface '' n Nov 12 20:56:09 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:56:09.274+0600 7f17dd634700 -1 osd.10 239617 set_numa_affinity unable to identify public interface '' n Nov 12 20:57:18 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:57:18.508+0600 7f17dd634700 -1 osd.10 239623 set_numa_affinity unable to identify public interface '' n Nov 12 20:58:27 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:58:27.650+0600 7f17d341a700 -1 osd.10 239629 _committed_osd_maps marked down 6 > osd_max_markdown_count Nov 12 20:58:27 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:58:27.654+0600 7f17e0e97700 -1 received signal: Interrupt from Kernel ( Could be generated by pthread_ Nov 12 20:58:27 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:58:27.654+0600 7f17e0e97700 -1 osd.10 239629 *** Got signal Interrupt *** Nov 12 20:58:27 ceph6.bol-online.com ceph-osd[66729]: 2023-11-12T20:58:27.654+0600 7f17e0e97700 -1 osd.10 239629 *** Immediate shutdown (osd_fast_shutdown=true) *** lines 1-15/15 (END) Regards Mosharaf Hossain Manager, Product Development IT Division Bangladesh Export Import Company Ltd. Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 Cell: +8801787680828, Email: mosharaf.hossain@xxxxxxxxxxxxxx, Web: www.bol-online.com <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A> On Sun, Nov 12, 2023 at 8:53 PM Mosharaf Hossain < mosharaf.hossain@xxxxxxxxxxxxxx> wrote: > Hello > THis osd got up but getting following error and it getting up and down. > > 2023-11-12T20:53:04.652+0600 7f17e3e9d700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f17c2dff700' had timed out after 15 > 2023-11-12T20:53:04.652+0600 7f17e3e9d700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f17c4602700' had timed out after 15 > 2023-11-12T20:53:04.652+0600 7f17e3e9d700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f17c5604700' had timed out after 15 > 2023-11-12T20:53:04.652+0600 7f17e3e9d700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f17c5e05700' had timed out after 15 > 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f17c05fa700' had timed out after 15 > 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f17c15fc700' had timed out after 15 > 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f17c1dfd700' had timed out after 15 > 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f17c2dff700' had timed out after 15 > 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f17c4602700' had timed out after 15 > 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f17c5604700' had timed out after 15 > 2023-11-12T20:53:04.652+0600 7f17e2e9b700 1 heartbeat_map is_healthy > 'OSD::osd_op_tp thread 0x7f17c5e05700' had timed out after 15 > > > > Regards > Mosharaf Hossain > Manager, Product Development > IT Division > > Bangladesh Export Import Company Ltd. > > Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh > > Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 > > Cell: +8801787680828, Email: mosharaf.hossain@xxxxxxxxxxxxxx, Web: > www.bol-online.com > > <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A> > > > > On Sun, Nov 12, 2023 at 8:51 PM Mosharaf Hossain < > mosharaf.hossain@xxxxxxxxxxxxxx> wrote: > >> Hello >> I am getting the following error for SSD osd.10. >> >> root@ceph6:~# tailf /var/log/ceph/ceph-osd.10.log >> 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 <cls> >> /build/ceph-15.2.17/src/cls/hello/cls_hello.cc:312: loading cls_hello >> 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 _get_class not permitted to >> load lua >> 2023-11-12T20:47:14.785+0600 7f17e9213d80 1 osd.10 239394 warning: got >> an error loading one or more classes: (1) Operation not permitted >> 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has >> features 432629239337189376, adjusting msgr requires for clients >> 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has >> features 432629239337189376 was 8705, adjusting msgr requires for mons >> 2023-11-12T20:47:14.785+0600 7f17e9213d80 0 osd.10 239394 crush map has >> features 3314933000854323200, adjusting msgr requires for osds >> 2023-11-12T20:47:14.785+0600 7f17e9213d80 1 osd.10 239394 >> check_osdmap_features require_osd_release unknown -> octopus >> 2023-11-12T20:47:22.942+0600 7f17e9213d80 0 osd.10 239394 load_pgs >> 2023-11-12T20:47:53.767+0600 7f17e9213d80 0 >> bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed >> for next, latency = 25923326091ns >> 2023-11-12T20:49:16.757+0600 7f17e9213d80 0 >> bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed >> for next, latency = 74835459564ns >> 2023-11-12T20:50:14.991+0600 7f17e9213d80 0 >> bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed >> for next, latency = 42822161884ns >> >> >> >> Regards >> Mosharaf Hossain >> Manager, Product Development >> IT Division >> >> Bangladesh Export Import Company Ltd. >> >> Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh >> >> Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 >> >> Cell: +8801787680828, Email: mosharaf.hossain@xxxxxxxxxxxxxx, Web: >> www.bol-online.com >> >> <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A> >> >> >> >> On Sun, Nov 12, 2023 at 8:43 PM Mosharaf Hossain < >> mosharaf.hossain@xxxxxxxxxxxxxx> wrote: >> >>> Hello >>> In my CEPH Nautilus cluster, some OSDs have unexpectedly gone offline, >>> despite all SSD disks appearing to be active on the node. What might be the >>> cause of this issue? >>> >>> #ceph osd tree [showing disk for ceph6 status] >>> -11 19.20854 host ceph6 >>> 10 ssd 1.74609 osd.10 down 0 1.00000 >>> 35 ssd 1.74609 osd.35 up 1.00000 1.00000 >>> 40 ssd 1.74609 osd.40 down 0 1.00000 >>> 45 ssd 1.74609 osd.45 down 0 1.00000 >>> 50 ssd 1.74609 osd.50 down 0 1.00000 >>> 54 ssd 1.74609 osd.54 up 1.00000 1.00000 >>> 58 ssd 1.74609 osd.58 up 1.00000 1.00000 >>> 59 ssd 1.74609 osd.59 up 1.00000 1.00000 >>> 67 ssd 1.74660 osd.67 up 1.00000 1.00000 >>> 68 ssd 1.74660 osd.68 down 0 1.00000 >>> 90 ssd 1.74660 osd.90 up 1.00000 1.00000 >>> >>> root@ceph6:~# fdisk -l | grep 1920383410176 >>> Disk /dev/sdb: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors >>> Disk /dev/sda: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors >>> Disk /dev/sdd: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors >>> Disk /dev/sdc: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors >>> Disk /dev/sdf: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors >>> Disk /dev/sdg: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors >>> Disk /dev/sdh: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors >>> Disk /dev/sdi: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors >>> Disk /dev/sde: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors >>> Disk /dev/sdj: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors >>> Disk /dev/sdk: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors >>> Disk /dev/sdl: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors >>> >>> >>> >>> Regards >>> Mosharaf Hossain >>> Manager, Product Development >>> IT Division >>> >>> Bangladesh Export Import Company Ltd. >>> >>> Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh >>> >>> Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757 >>> >>> Cell: +8801787680828, Email: mosharaf.hossain@xxxxxxxxxxxxxx, Web: >>> www.bol-online.com >>> >>> <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A> >>> >>> _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |