On 05/17/2012 06:54 AM, Voznesensky Vladimir wrote: > Hello. > > It seems that corosync gives a very long latency on a one-node > configuration. > We have developed a small test passing messages between two threads. > Each message carried an original tsc timestamp counter value, so we have > been able to compute the difference with the receiver's tsc. > > 1 (100-bytes) message pass took about 200 us. > 30000 messages batch gave about 2 ms. > > Some ipc ring-buffer implementation using eventfd showed us less than a > 1 microsecond between two processor cores to pass a 128 byte message in > a 1000 messages batch. > > So, what's the source of such relatively high latency? > > Thank you. > Totem interaction is introducing the latency, and the figures you quote, 200 usec, seem quite good to me. The TLDR version is: In order to transmit messages, our protocol (Totem) must posses a token. The token is rotated in a ring of processors. When there is one processor, the token rotates on the same processor. In the case of 30,000 100 byte messages, this would translate into about 2142 totem messages that are sent over the wire. A processor may only send a certain number of messages per token rotation, as dictated by the flow control parameters (see man corosync.conf look for window). 2142 = (100 * 30,000) / 1400 1400 = mtu 100 = message size 30000 = number of messages The act of sending and receiving the token introduces small latencies, which add up over the 126 token rotations your example might give. 126 = 2142 / 17 2142 = messages 17 = max messages per rotation It would make more sense for single node conditions to short-circuit totem altogether, and I believe the upcoming roadmap describes this exact feature. Regards -steve > - VV > _______________________________________________ > discuss mailing list > discuss@xxxxxxxxxxxx > http://lists.corosync.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss@xxxxxxxxxxxx http://lists.corosync.org/mailman/listinfo/discuss
