Hi Steven Dake, We have tested your patch last week for a long time, but I am so sorry to tell you that we could not duplicate the issue again---it never run into "FAIL" state. The corosync runs so strongly beyond all our expectation. It's both a good news and a bad news :( Maybe there is unknown condition to duplicate this issue, and I will continue to monitor it. On Wed, Oct 12, 2011 at 8:24 PM, Yunkai Zhang <qiushu.zyk@xxxxxxxxxx> wrote: > Hi Steven Dake, > > Thank you for your reply. > I would change this discussion CC to discuss@xxxxxxxxxxxx instead of > openais@xxxxxxxxxxxxxx > as we known openais@xxxxxxxxxxxxxx could not work. > > On Tue, Oct 11, 2011 at 10:54 PM, Steven Dake <sdake@xxxxxxxxxx> wrote: >> On 10/10/2011 08:58 PM, qiushu.zyk@xxxxxxxxxx wrote: >>> From: Yunkai Zhang <qiushu.zyk@xxxxxxxxxx> >>> >>> In our 20 nodes cluster testing(corosync v1.4.1 vs sheepdog), an abormal exit >>> would occur when consensus timeout expired and if there was no other processors >>> in consensus_list. >>> >>> = analysis = >>> 1. when consenus timeout, corosync would enter memb_state_consensus_timeout_expired >>> function. >>> >>> 2. if its consensus_list only contain my_id, the code would >>> execute memb_set_merge which make my_failed_list equal to >>> my_proc_list. >>> >>> 3. call memb_state_gather_enter function and mcast a join message which >>> contain proc_list and failed_list with the same processor IDs. >>> >>> 4. the join message would be received by itself and >>> memb_join_process/memb_consensus_agreed would by called. >>> >>> 5. because the proc_list equal to failed_list in the join message, >>> the assert instruction will be reached, then an abnormal exit occur. >>> >>> = solution = >>> This patch try to resolve this issue by remove my_id from >>> my_failed_list before corosync call memb_state_gather_enter from >>> memb_state_consensus_timeout_expired. >>> >>> when network partition occur and the processor can't communicate with >>> all other processors, it will form an single ring with only itself other >>> than trigger abnormal exit. >>> >> >> Yunkai, >> >> Thank you for the patch. I am really hesitant to make any changes to >> totemsrp that I haven't thought long and hard about. Your solution is >> clever and well thought out, but totem has thousands of details - to the >> point that I always want to get to the root cause of the issue when >> fixing problems. >> >> I think your running into a "FAILED TO RECV" state. There is a patch >> outstanding for this issue but we have been unable to find anyone to >> test it. Our environments don't demonstrate a failed to receive scenario. >> >> Can you verify if you have FAILED TO RECV (should be in the fplay data). >> > > Yes, corosync will run into a "FAILED TO RECV" state in > message_handler_orf_token function. > We can see the last three lines of logging messages from > /var/log/cluster/corosync.log as following: > > Aug 25 13:42:25 corosync [TOTEM ] FAILED TO RECEIVE > Aug 25 13:42:25 corosync [TOTEM ] entering GATHER state from 6. > Aug 25 13:42:27 corosync [TOTEM ] entering GATHER state from 0. > > According our testing, we can duplicate this issue so easily with > two conditions: > 1). more than 20 nodes (not exactly). > 2). increasing the network load which will cause broadcasting > message(regular msg/join msg) failed frequently. > >> If so, can you run with the patch: >> https://bugzilla.redhat.com/show_bug.cgi?id=636583 comment #8 > > Thank for your patch, > I plan to test it next week as I have no enough nodes to test now. > >> >> Thanks! >> -steve >> >> >>> Signed-off-by: Yunkai Zhang <qiushu.zyk@xxxxxxxxxx> >>> --- >>> exec/totemsrp.c | 33 +++++++++++++++++++++++++++++++++ >>> 1 files changed, 33 insertions(+), 0 deletions(-) >>> >>> diff --git a/exec/totemsrp.c b/exec/totemsrp.c >>> index 0778d55..653f801 100644 >>> --- a/exec/totemsrp.c >>> +++ b/exec/totemsrp.c >>> @@ -1324,6 +1324,33 @@ static void memb_set_merge ( >>> return; >>> } >>> >>> +/* >>> + * remove subset from fullset >>> + */ >>> +static void memb_set_remove( >>> + const struct srp_addr *subset, int subset_entries, >>> + struct srp_addr *fullset, int *fullset_entries) >>> +{ >>> + int found = 0; >>> + int i; >>> + int j; >>> + >>> + for (i = 0; i < subset_entries; i++) { >>> + for (j = 0; j < *fullset_entries; j++) { >>> + if (srp_addr_equal (&fullset[j], &subset[i])) { >>> + found = 1; >>> + break; >>> + } >>> + } >>> + if (found == 1) { >>> + for (; j < (*fullset_entries-1); j++) { >>> + srp_addr_copy (&fullset[j], &fullset[j+1]); >>> + } >>> + *fullset_entries = *fullset_entries - 1; >>> + } >>> + } >>> +} >>> + >>> static void memb_set_and_with_ring_id ( >>> struct srp_addr *set1, >>> struct memb_ring_id *set1_ring_ids, >>> @@ -1541,6 +1568,12 @@ static void memb_state_consensus_timeout_expired ( >>> >>> memb_set_merge (no_consensus_list, no_consensus_list_entries, >>> instance->my_failed_list, &instance->my_failed_list_entries); >>> + >>> + if (instance->my_proc_list_entries == instance->my_failed_list_entries){ >>> + memb_set_remove (&instance->my_id, 1, >>> + instance->my_failed_list, &instance->my_failed_list_entries); >>> + } >>> + >>> memb_state_gather_enter (instance, 0); >>> } >>> } >> >> > > > > -- > Yunkai Zhang > work at taobao.com > -- Yunkai Zhang Work at Taobao _______________________________________________ discuss mailing list discuss@xxxxxxxxxxxx http://lists.corosync.org/mailman/listinfo/discuss
