Re: ib_post_send execution time

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Oct 24, 2014 at 3:39 AM, Eli Cohen <eli@xxxxxxxxxxxxxxxxxx> wrote:
> On Thu, Oct 23, 2014 at 11:45:05AM -0700, Roland Dreier wrote:
>> On Thu, Oct 23, 2014 at 10:21 AM, Evgenii Smirnov
>> <evgenii.smirnov@xxxxxxxxxxxxxxxx> wrote:
>> > I am trying to achieve high packet per second throughput with 2-byte
>> > messages over Infiniband from kernel using IB_SEND verb. The most I
>> > can get so far is 3.5 Mpps. However, ib_send_bw utility from perftest
>> > package is able to send 2-byte packets with rate of 9 Mpps.
>> > After some profiling I found that execution of ib_post_send function
>> > in kernel takes about 213 ns in average, for the user-space function
>> > ibv_post_send takes only about 57 ns.
>> > As I understand, these functions do almost same operations. The work
>> > request fields and queue pair parameters are also the same. Why do
>> > they have such big difference in execution times?
>>
>>
>> Interesting.  I guess it would be useful to look at perf top / and or
>> get a perf report with "perf report -a -g" when running your high PPS
>> workload, and see where the time is wasted.
>>
>
> I assume ib_send_bw uses inline with blueflame so it may be part of
> the explanation to the differences you see.

I think it should be the other way around... when we use inline we
consume more CPU cycles and here we see notable different (213ns --
kernel 57ns user) in favor of libmlx4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Photo]     [Yosemite News]     [Yosemite Photos]     [Linux Kernel]     [Linux SCSI]     [XFree86]
  Powered by Linux