Re: OSD disk is active in node but ceph show osd down and out

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello
THis osd got up but getting following error and it getting up and down.

2023-11-12T20:53:04.652+0600 7f17e3e9d700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f17c2dff700' had timed out after 15
2023-11-12T20:53:04.652+0600 7f17e3e9d700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f17c4602700' had timed out after 15
2023-11-12T20:53:04.652+0600 7f17e3e9d700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f17c5604700' had timed out after 15
2023-11-12T20:53:04.652+0600 7f17e3e9d700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f17c5e05700' had timed out after 15
2023-11-12T20:53:04.652+0600 7f17e2e9b700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f17c05fa700' had timed out after 15
2023-11-12T20:53:04.652+0600 7f17e2e9b700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f17c15fc700' had timed out after 15
2023-11-12T20:53:04.652+0600 7f17e2e9b700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f17c1dfd700' had timed out after 15
2023-11-12T20:53:04.652+0600 7f17e2e9b700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f17c2dff700' had timed out after 15
2023-11-12T20:53:04.652+0600 7f17e2e9b700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f17c4602700' had timed out after 15
2023-11-12T20:53:04.652+0600 7f17e2e9b700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f17c5604700' had timed out after 15
2023-11-12T20:53:04.652+0600 7f17e2e9b700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f17c5e05700' had timed out after 15



Regards
Mosharaf Hossain
Manager, Product Development
IT Division

Bangladesh Export Import Company Ltd.

Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh

Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757

Cell: +8801787680828, Email: mosharaf.hossain@xxxxxxxxxxxxxx, Web:
www.bol-online.com
<https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A>



On Sun, Nov 12, 2023 at 8:51 PM Mosharaf Hossain <
mosharaf.hossain@xxxxxxxxxxxxxx> wrote:

> Hello
> I am getting the following error for SSD osd.10.
>
> root@ceph6:~# tailf /var/log/ceph/ceph-osd.10.log
> 2023-11-12T20:47:14.785+0600 7f17e9213d80  0 <cls>
> /build/ceph-15.2.17/src/cls/hello/cls_hello.cc:312: loading cls_hello
> 2023-11-12T20:47:14.785+0600 7f17e9213d80  0 _get_class not permitted to
> load lua
> 2023-11-12T20:47:14.785+0600 7f17e9213d80  1 osd.10 239394 warning: got an
> error loading one or more classes: (1) Operation not permitted
> 2023-11-12T20:47:14.785+0600 7f17e9213d80  0 osd.10 239394 crush map has
> features 432629239337189376, adjusting msgr requires for clients
> 2023-11-12T20:47:14.785+0600 7f17e9213d80  0 osd.10 239394 crush map has
> features 432629239337189376 was 8705, adjusting msgr requires for mons
> 2023-11-12T20:47:14.785+0600 7f17e9213d80  0 osd.10 239394 crush map has
> features 3314933000854323200, adjusting msgr requires for osds
> 2023-11-12T20:47:14.785+0600 7f17e9213d80  1 osd.10 239394
> check_osdmap_features require_osd_release unknown -> octopus
> 2023-11-12T20:47:22.942+0600 7f17e9213d80  0 osd.10 239394 load_pgs
> 2023-11-12T20:47:53.767+0600 7f17e9213d80  0
> bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed
> for next, latency = 25923326091ns
> 2023-11-12T20:49:16.757+0600 7f17e9213d80  0
> bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed
> for next, latency = 74835459564ns
> 2023-11-12T20:50:14.991+0600 7f17e9213d80  0
> bluestore(/var/lib/ceph/osd/ceph-10) log_latency slow operation observed
> for next, latency = 42822161884ns
>
>
>
> Regards
> Mosharaf Hossain
> Manager, Product Development
> IT Division
>
> Bangladesh Export Import Company Ltd.
>
> Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
>
> Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
>
> Cell: +8801787680828, Email: mosharaf.hossain@xxxxxxxxxxxxxx, Web:
> www.bol-online.com
>
> <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A>
>
>
>
> On Sun, Nov 12, 2023 at 8:43 PM Mosharaf Hossain <
> mosharaf.hossain@xxxxxxxxxxxxxx> wrote:
>
>> Hello
>> In my CEPH Nautilus cluster, some OSDs have unexpectedly gone offline,
>> despite all SSD disks appearing to be active on the node. What might be the
>> cause of this issue?
>>
>> #ceph osd tree [showing disk for ceph6 status]
>> -11          19.20854      host ceph6
>>  10    ssd    1.74609          osd.10     down         0  1.00000
>>  35    ssd    1.74609          osd.35       up   1.00000  1.00000
>>  40    ssd    1.74609          osd.40     down         0  1.00000
>>  45    ssd    1.74609          osd.45     down         0  1.00000
>>  50    ssd    1.74609          osd.50     down         0  1.00000
>>  54    ssd    1.74609          osd.54       up   1.00000  1.00000
>>  58    ssd    1.74609          osd.58       up   1.00000  1.00000
>>  59    ssd    1.74609          osd.59       up   1.00000  1.00000
>>  67    ssd    1.74660          osd.67       up   1.00000  1.00000
>>  68    ssd    1.74660          osd.68     down         0  1.00000
>>  90    ssd    1.74660          osd.90       up   1.00000  1.00000
>>
>> root@ceph6:~# fdisk -l | grep 1920383410176
>> Disk /dev/sdb: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
>> Disk /dev/sda: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
>> Disk /dev/sdd: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
>> Disk /dev/sdc: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
>> Disk /dev/sdf: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
>> Disk /dev/sdg: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
>> Disk /dev/sdh: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
>> Disk /dev/sdi: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
>> Disk /dev/sde: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
>> Disk /dev/sdj: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
>> Disk /dev/sdk: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
>> Disk /dev/sdl: 1.8 TiB, 1920383410176 bytes, 3750748848 sectors
>>
>>
>>
>> Regards
>> Mosharaf Hossain
>> Manager, Product Development
>> IT Division
>>
>> Bangladesh Export Import Company Ltd.
>>
>> Level-8, SAM Tower, Plot #4, Road #22, Gulshan-1, Dhaka-1212,Bangladesh
>>
>> Tel: +880 9609 000 999, +880 2 5881 5559, Ext: 14191, Fax: +880 2 9895757
>>
>> Cell: +8801787680828, Email: mosharaf.hossain@xxxxxxxxxxxxxx, Web:
>> www.bol-online.com
>>
>> <https://www.google.com/url?q=http://www.bol-online.com&sa=D&source=hangouts&ust=1557908951423000&usg=AFQjCNGMxIuHSHsD3qO6y5JddpEZ0S592A>
>>
>>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux