Hi Honza,
Thanks for the clarifications.
Just curios. Why are you doing that? Is there something you are missing
in CPG? If so, can you please try to describe what is that? Because if
that functionality is generally useful, we can consider implement it to
CPG (or create new service).
There are some minor usage issues with CPG service, for example, it uses
pid as the memberId, but apps might want to specify their own memberId
to join the groups; but this is not the main reason for me to go on this
path.
We are trying to use corosync in an unusual way. We are creating an
hierarchical cluster (cluster of chassis) that interact with each other.
We want to use corosync to form this cluster of clusters. So we would
want only one instance of corosync running per chassis, but we want apps
running on different blades of the chassis to be able to talk to the
daemon running on a single blade. This cannot be achieved unless we
change QB to support TCP as a transport and the cpg library can talk to
cpg service through TCP transport.
I considered enhancing QB for this support, but it seems like a lot more
work. So the approach we decided to take is that;
- We already have an IPC library for our applications to talk to each other.
- So we are converting one of our app as a corosync-service which
implements CPG functionality and can provide the same functionality to
our apps.
Corosync is basically state machine with following states: Operational,
Gather, Recovery. Sync service more or less add new state, syncing (even
from totem perspective, it's Operational state).
When corosync membership changed (new processor appeared or some
processor(s) disappeared), corosync starts gathering, then recovery and
hopefully ends in operational state. Then, sync_init is called (with
list of processors). It is usually used for saving membership. Right
after that, sync_process is called. Sync_process can return -1, when it
wishes to be called again, or 0 if not. Sync_process is protected by
barrier, so until ALL messages sent in sync_init/sync_process are
delivered to ALL nodes, it doesn't continue. When ALL messages from ALL
processors for given services are delivered, sync_activate is called
(this is correct place where to inform IPC clients about membership
change). Last one is sync_abort. This is called when new node appeared
or some node(s) disappeared during sync_process. If that happen, whole
sync process repeats (so sync_init is called, then sync_process, ...).
So does it mean that, any other messages sent by the services while sync
is happening, wont be sent until sync completes?
For example;
At T1 -- App sends a message for a CPG group
T2 -- CPG service receives the message and is about to send it over
totem ring to all the other services on the ring
T3 -- Before service could send the message over the ring, it
receives a sync_init/sync_process and it starts sending messages to
other peers as part of synchronization.
T4 -- While sync is in progress, it sends the message over the ring
So in the above scenario, does corosync delay sending the group messages
while service is sending synchronize messages? How does it differentiate
between these messages?
The use case for the above message sequence is that;
-- When a new node joins the ring and the CPG service is started on the
new node, all the other existing CPG services would want to let the new
node know about the existing groups and the group membership. So lets
say that CPG services are sending the updates to the new service about
their group membership and if they receive a groupLeave (for ex, the app
crashes), and they send a leave message to all the CPG services over
totem, we must ensure the proper ordering of the group updates and the
leave messages. Otherwise the services will go out of sync.
Regards,
Shridhar
_______________________________________________
discuss mailing list
discuss@xxxxxxxxxxxx
http://lists.corosync.org/mailman/listinfo/discuss
