On 03/19/2013 03:18 AM, Guangliang Zhao wrote:
Hi list, There is a issue when I tested corosync(v1.4.5) with 11 nodes. I am not very familiar with the corosync, so please correct me if I am wrong. The steps are following: 1.Make sure the corosync debug is off 2.Start openais on every node, and all of them are ok. 3.Stop openais on 5 nodes, it takes so longe time, and the retransmit list started growing. I got a piece of log from one node via corosync-blackbox: rec=[79224] Tracing(1) Messsage=Received ringid(192.168.100.1:6564) seq 1fd rec=[79225] Tracing(1) Messsage=Delivering 1fc to 1fd rec=[79226] Tracing(1) Messsage=Delivering MCAST message with seq 1fd to pending delivery queue rec=[79227] Tracing(1) Messsage=releasing messages up to and including 1fb rec=[79228] Tracing(1) Messsage=releasing messages up to and including 1fd rec=[79229] Log Message=got quorate request on 0x6d0980 rec=[79230] Log Message=got quorate request on 0x6d0980 rec=[79231] Log Message=Retransmit List 1 rec=[79232] Log Message=Retransmit List: 201 rec=[79233] Tracing(1) Messsage=mcasted message added to pending queue rec=[79234] Log Message=Retransmit List 1 rec=[79235] Log Message=Retransmit List: 201 rec=[79236] Tracing(1) Messsage=Delivering 1fd to 205 rec=[79237] Tracing(1) Messsage=Received ringid(192.168.100.1:6564) seq 205 rec=[79238] Tracing(1) Messsage=Delivering 1fd to 205 rec=[79239] Log Message=Retransmit List 1 rec=[79240] Log Message=Retransmit List: 201 rec=[79241] Tracing(1) Messsage=Delivering 1fd to 205 rec=[79242] Log Message=Retransmit List 2 rec=[79243] Log Message=Retransmit List: 201 202 rec=[79244] Tracing(1) Messsage=Delivering 1fd to 205 rec=[79245] Log Message=Retransmit List 2 rec=[79246] Log Message=Retransmit List: 201 202 There is a piece of code in exec/totemsrp.c: 3775 if (range) { 3776 TRACE1 ("Delivering %x to %x\n", instance->my_high_delivered, 3777 end_point); 3778 } ... 3785 for (i = 1; i <= range; i++) { 3786 3787 void *ptr = 0; 3788 3789 /* 3790 * If out of range of sort queue, stop assembly 3791 */ 3792 res = sq_in_range (&instance->regular_sort_queue, 3793 my_high_delivered_stored + i); 3794 if (res == 0) { 3795 break; 3796 } 3797 3798 res = sq_item_get (&instance->regular_sort_queue, 3799 my_high_delivered_stored + i, &ptr); 3800 /* 3801 * If hole, stop assembly 3802 */ 3803 if (res != 0 && skip == 0) { 3804 break; 3805 } 3806 3807 instance->my_high_delivered = my_high_delivered_stored + i; ... 3841 /* 3842 * Message found 3843 */ 3844 TRACE1 ("Delivering MCAST message with seq %x to pending delivery queue\n", 3845 mcast_header.seq); From these log and code, We could know that the message 1fe 1ff 200 have not been delivered and it should jump out of the loop through the two break sentences. The first if only check the seq id range, and the second one should be the most suspect. include/corosync/sq.h: 264 static inline unsigned int sq_item_get ( 265 const struct sq *sq, 266 unsigned int seq_id, 267 void **sq_item_out) ... 286 if (sq->items_inuse[sq_position] == 0) { 287 return (ENOENT); 288 }I think the items_inuse array maybe cleared sometimes, and it return 0 when we access it. However, I couldn't study deep in more, so could anyone give me some hints?
items_inuse[sq_position] should contain zero if there is no entry. If there is no entry, we want to stop processing in the above code because it is a hole in the messages.
The sort queue is a circular array which is cleared as sq_item_release is called. This should only occur after the message has been delivered to all nodes on the ring in totemsrp.c:messages_free.
Regards -steve _______________________________________________ discuss mailing list discuss@xxxxxxxxxxxx http://lists.corosync.org/mailman/listinfo/discuss
