Re: Luminous: resilience - private interface down , no read/write

[David Turner <drakonstein@xxxxxxxxx>]

How many disks in each node? 68? If yes, then change it to 69. Also running with ec 4+1 is bad for the same reason as running with size=2 min_size=1 which has been mentioned and discussed multiple times on the ML.

On Wed, May 23, 2018, 3:39 AM nokia ceph <nokiacephusers@xxxxxxxxx> wrote:

Hi David Turner,

This is our ceph config under mon section , we have EC 4+1 and set the failure domain as host and osd_min_down_reporters to 4 ( osds from 4 different host ) .

[mon]

mon_compact_on_start = True

mon_osd_down_out_interval = 86400

mon_osd_down_out_subtree_limit = host

mon_osd_min_down_reporters = 4

mon_osd_reporter_subtree_level = host

We have 68 disks , can we increase  sd_min_down_reporters  to 68 ?

Thanks,

Muthu

On Tue, May 22, 2018 at 5:46 PM, David Turner <drakonstein@xxxxxxxxx> wrote:

What happens when a storage node loses its cluster network but not it's public network is that all other osss on the cluster see that it's down and report that to the mons, but the node call still talk to the mons telling the mons that it is up and in fact everything else is down.

The setting osd _min_reporters (I think that's the name of it off the top of my head) is designed to help with this scenario. It's default is 1 which means any osd on either side of the network problem will be trusted by the mons to mark osds down. What you want to do with this seeing is to set it to at least 1 more than the number of osds in your failure domain. If the failure domain is host and each node has 32 osds, then setting it to 33 will prevent a full problematic node from being able to cause havoc.

The osds will still try to mark themselves as up and this will still cause problems for read until the osd process stops or the network comes back up. There might be a seeing for how long an odd will try telling the mons it's up, but this isn't really a situation I've come across after initial testing and installation of nodes.

On Tue, May 22, 2018, 1:47 AM nokia ceph <nokiacephusers@xxxxxxxxx> wrote:

Hi Ceph users,

We have a cluster with 5 node (67 disks) and EC 4+1 configuration and min_size set as 4.

Ceph version : 12.2.5

While executing one of our resilience usecase , making private interface down on one of the node, till kraken we saw less outage in rados (60s) .

Now with luminous, we could able to see rados read/write outage for more than 200s . In the logs we could able to see that peer OSDs inform that one of the node OSDs are down however the OSDs  defend like it is wrongly marked down and does not move to down state for long time.

2018-05-22 05:37:17.871049 7f6ac71e6700  0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.1 down, but it is still running

2018-05-22 05:37:17.871072 7f6ac71e6700  0 log_channel(cluster) log [DBG] : map e35690 wrongly marked me down at e35689

2018-05-22 05:37:17.878347 7f6ac71e6700  0 osd.1 35690 crush map has features 1009107927421960192, adjusting msgr requires for osds

2018-05-22 05:37:18.296643 7f6ac71e6700  0 osd.1 35691 crush map has features 1009107927421960192, adjusting msgr requires for osds

Only when all 67 OSDs are move to down state , the read/write traffic is resumed.

Could you please help us in resolving this issue and if it is bug , we will create corresponding ticket.

Thanks,

Muthu

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com