On Tue, Mar 19, 2013 at 07:44:21AM -0700, Steven Dake wrote:
> On 03/19/2013 03:18 AM, Guangliang Zhao wrote:
> >Hi list,
Hi Steven,
Thanks for your reply.
> >
> >There is a issue when I tested corosync(v1.4.5) with 11 nodes. I am not very familiar with the corosync, so please correct me if I am wrong. The steps are following:
> >
> >1.Make sure the corosync debug is off
> >2.Start openais on every node, and all of them are ok.
> >3.Stop openais on 5 nodes, it takes so longe time, and the retransmit list started growing.
> >
> >I got a piece of log from one node via corosync-blackbox:
> >
> >rec=[79224] Tracing(1) Messsage=Received ringid(192.168.100.1:6564) seq 1fd
> >rec=[79225] Tracing(1) Messsage=Delivering 1fc to 1fd
> >rec=[79226] Tracing(1) Messsage=Delivering MCAST message with seq 1fd to pending delivery queue
> >rec=[79227] Tracing(1) Messsage=releasing messages up to and including 1fb
> >rec=[79228] Tracing(1) Messsage=releasing messages up to and including 1fd
> >rec=[79229] Log Message=got quorate request on 0x6d0980
> >rec=[79230] Log Message=got quorate request on 0x6d0980
> >rec=[79231] Log Message=Retransmit List 1
> >rec=[79232] Log Message=Retransmit List: 201
> >rec=[79233] Tracing(1) Messsage=mcasted message added to pending queue
> >rec=[79234] Log Message=Retransmit List 1
> >rec=[79235] Log Message=Retransmit List: 201
> >rec=[79236] Tracing(1) Messsage=Delivering 1fd to 205
> >rec=[79237] Tracing(1) Messsage=Received ringid(192.168.100.1:6564) seq 205
> >rec=[79238] Tracing(1) Messsage=Delivering 1fd to 205
> >rec=[79239] Log Message=Retransmit List 1
> >rec=[79240] Log Message=Retransmit List: 201
> >rec=[79241] Tracing(1) Messsage=Delivering 1fd to 205
> >rec=[79242] Log Message=Retransmit List 2
> >rec=[79243] Log Message=Retransmit List: 201 202
> >rec=[79244] Tracing(1) Messsage=Delivering 1fd to 205
> >rec=[79245] Log Message=Retransmit List 2
> >rec=[79246] Log Message=Retransmit List: 201 202
> >
> >There is a piece of code in exec/totemsrp.c:
> >
> >3775 if (range) {
> >3776 TRACE1 ("Delivering %x to %x\n", instance->my_high_delivered,
> >3777 end_point);
> >3778 }
> >
> >...
> >
> >3785 for (i = 1; i <= range; i++) {
> >3786
> >3787 void *ptr = 0;
> >3788
> >3789 /*
> >3790 * If out of range of sort queue, stop assembly
> >3791 */
> >3792 res = sq_in_range (&instance->regular_sort_queue,
> >3793 my_high_delivered_stored + i);
> >3794 if (res == 0) {
> >3795 break;
> >3796 }
> >3797
> >3798 res = sq_item_get (&instance->regular_sort_queue,
> >3799 my_high_delivered_stored + i, &ptr);
> >3800 /*
> >3801 * If hole, stop assembly
> >3802 */
> >3803 if (res != 0 && skip == 0) {
> >3804 break;
> >3805 }
> >3806
> >3807 instance->my_high_delivered = my_high_delivered_stored + i;
> >
> >...
> >
> >3841 /*
> >3842 * Message found
> >3843 */
> >3844 TRACE1 ("Delivering MCAST message with seq %x to pending delivery queue\n",
> >3845 mcast_header.seq);
> >
> > From these log and code, We could know that the message 1fe 1ff 200 have not been delivered and it should jump out of the loop through the two break sentences.
> >
> >The first if only check the seq id range, and the second one should be the most suspect.
> >
> >include/corosync/sq.h:
> >
> >264 static inline unsigned int sq_item_get (
> >265 const struct sq *sq,
> >266 unsigned int seq_id,
> >267 void **sq_item_out)
> >
> >...
> >
> >286 if (sq->items_inuse[sq_position] == 0) {
> >287 return (ENOENT);
> >288 }
> >I think the items_inuse array maybe cleared sometimes, and it return 0 when we access it. However, I couldn't study deep in more, so could anyone give me some hints?
> >
>
> items_inuse[sq_position] should contain zero if there is no entry.
> If there is no entry, we want to stop processing in the above code
> because it is a hole in the messages.
If we want skip the hole in the messages, I think the my_high_delivered
or more parameters should be updated, but didn't, so it always try to
deliver the messages from my_high_delivered + 1, but couldn't success,
because the my_high_delivered + 1 message is a hole?
I collected the result of corosync-blackbox from one of the nodes, but it is a
pretty big log, I would add it as an attachment next mail if you need.
>
> The sort queue is a circular array which is cleared as
> sq_item_release is called. This should only occur after the message
> has been delivered to all nodes on the ring in
> totemsrp.c:messages_free.
>
> Regards
> -steve
>
--
Best regards,
Guangliang
_______________________________________________
discuss mailing list
discuss@xxxxxxxxxxxx
http://lists.corosync.org/mailman/listinfo/discuss
