Re: Question about message generation/origination during SYNC

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



You might want to post this to the clusterlabs mailing list. This list
is being phased out, so there probably aren't many people left here.

http://clusterlabs.org/mailman/listinfo

digimer

On 19/06/15 10:39 PM, jason wrote:
>  Hi All,
> 
> I have made two patches (attached) to prove this issue really exist.
> 
> In prove_cfg_can_send_messages_through_totem_during_sync.patch, it
> first comments out the functionality of ipc_allow_connections to make
> the first sync procedure the same as its subsequents, thus, make
> ipc_allow_connections always true. Then it lets cpg_sync_process() to
> always returns -1 in order to let sync do not stop. So with this
> patch, during corosync startup in a simple one node configuration, one
> can run corosync-cfgtool -R during sync. The result is that cfg
> command CAN pass through ipc and totem to send reload message out then
> do reload. So this patch can prove that cfg(library) can send messages
> through totem during sync without any restriction.
> 
> In simualte_a_cfg_message_during_sync_which_breaks_defrag.patch(do not
> need to base on the first one), it simulates a cfg message(named as
> breaker message in patch), and sends it right after sending every sync
> barrier messages. So the breaker message which is sent after the last
> sync barrier message(which cause sync_synchronization_completed())
> will be sent in the sync procedure but be received by
> assembly_list_inuse/assembly_list_free, not the expected
> assembly_list_inuse_trans/assembly_list_free_trans, and will finally
> cause a  "fragmented continuation %u is not equal to assembly
> last_frag_num..." log. I also place a assert(0) after that log in this
> patch to make it easy to see.  This can be reporduced by patching this
> patch and simply run a single node configuration.
> 
> I hope these two patches can make it easy to illustrate this issue.
> 
> On Fri, Jun 5, 2015 at 12:05 AM, jason <huzhijiang@xxxxxxxxx> wrote:
>> Please consider the possibility of this potential issue. As I
>> understand, things may happen like this:
>> a two-node cluster, with Node A and B.
>>
>> Node A                    Node B
>>                                1) Send the last SYNC message(seq n)
>>                                2) Send the token
>> 3) Got token
>> 4) Lost the last SYNC message(seq n)
>> 5) SYNC can not be done, need wait retransmit
>> 6) Originate a cfg message(seq n + 1) based on *trans* assembly buffer
>> 7) Can not immediatly delivery cfg message (seq n + 1) based on trans
>> assembly buffer
>> 8) Send the token
>>                                9) Got token
>>                                10) retransmit last SYNC message(seq n)
>>                                11) Send the token
>> 12) Got token
>> 13) Received the last SYNC message(seq n)
>> 14) SYNC done, switch to the normal assembly buffer
>> 15) Delivery cfg message(seq n + 1) based on *normal* assembly buffer! BAD!
>>
>> In this case, cfg message(seq n + 1) will corrupts our *normal* assembly buffer.
>>
>> Because corosync_sending_allowed() (called by cs_ipcs_msg_process())
>> returns QB_TRUE for those IPC connections who are
>> CS_LIB_FLOW_CONTROL_NOT_REQUIRED, such as cfg. So cfg message can
>> still pass IPC to totem during SYNC, thus, cause the step 6) to
>> happen.
>>
>> One straight way to solve this issue is changing all
>> CS_LIB_FLOW_CONTROL_NOT_REQUIRED to CS_LIB_FLOW_CONTROL_REQUIRED for
>> cfg service.
>>
>>
>> ---------- Forwarded message ----------
>> From: jason <huzhijiang@xxxxxxxxx>
>> Date: Fri, Jan 30, 2015 at 7:29 PM
>> Subject: Question about message generation/origination during SYNC
>> To: "discuss@xxxxxxxxxxxx" <discuss@xxxxxxxxxxxx>
>>
>>
>> Dear All,
>>
>> By analyzing current corosync code, I found that if some messages can
>> be generated from library during SYNC processing(such as
>> reload/shutdown over cfgtool, because they are
>> CS_LIB_FLOW_CONTROL_NOT_REQUIRED), or in other words, if they can be
>> generated on new_message_queue_trans queue because
>> instance->waiting_trans_ack was set to 1, then they may have chance to
>> be originated after the last SYNC message. In this situation, they
>> will be delivered after instance->waiting_trans_ack and
>> totempg_waiting_transack set back to 0, then the assembly for the
>> normal messages will be used to defrage  those messages, not the
>> expected assembly for the trans messages. This may finally result in
>> lost normal messages due to fragment number is not equal to assembly
>> last_frag_num.
>>
>> Please have a look if this really a problem or I have missed something?
>>
>> Thank you!
>>
>>
>>
>> --
>> Yours,
>> Jason
>>
>>
>> --
>> Yours,
>> Jason
> 
> 
> 


-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
_______________________________________________
discuss mailing list
discuss@xxxxxxxxxxxx
http://lists.corosync.org/mailman/listinfo/discuss



[Index of Archives]     [Linux Clusters]     [Corosync Project]     [Linux USB Devel]     [Linux Audio Users]     [Photo]     [Yosemite News]    [Yosemite Photos]    [Linux Kernel]     [Linux SCSI]     [X.Org]

  Powered by Linux