On 22/02/2021 17:55, Jason Gunthorpe wrote: > On Mon, Feb 22, 2021 at 05:36:17PM +0200, Gal Pressman wrote: > >> "Mellanox HCAs keep track of the last index for which the user received an >> event. Using this index, it is guaranteed that an event is generated immediately >> when a request completion notification is performed and a CQE has already been >> reported." > > I don't think verbs exposes this behavior. So in theory this hardware could generate events that the user app doesn't expect? >> This also sounds weird, why is an event generated for a completion that has >> already been reported? > > It eleminates races, if the consumer says 'I read up to X send me an > interrupt if X+1 exists' when X+1 already exists if there is a race > producer has already written it. So send an interrupt. Right, that's what I was getting at in my first question, this isn't the next completion from the device's perspective. So in such case the consumer index in the arm doorbell is used to indicate what should be considered a "new" completion? This is new from the app perspective. But looking at ibv_ud_pingpong for example, I don't understand how that could even work. The test arms the CQ on creation (consumder index 0), calls ibv_get_cq_event(), wakes up and immediately arms the CQ again (before polling, consumer index is still 0). This means that the next ibv_get_cq_event() will wake up immediately, as the CQ was armed twice with the same consumer index and the first completion already exists. Surely that's not what's meant to happen? >> So from my understanding of how this should work, the following code in perftest >> (ib_send_bw test) is buggy?: >> https://github.com/linux-rdma/perftest/blob/master/src/perftest_resources.c#L2955 >> >> Running this with 32 iterations, the client does something like: >> - arm cq >> - post send x 32 >> - wait for cq event >> - arm cq >> - poll cq (once, with batch size of 16) >> - no more post send (reached tot_iters) >> - wait for cq event (but an event has already been generated?) > > I don't know much about perf-test, but in verbs arming a non-empty CQ > is asking for trouble Do you have a way to verify whether this test gets stuck? Maybe I am missing something? What do you mean by arming a non-empty CQ? The man pages suggest a scheme where the app should call ibv_get_cq_event() followed by an ibv_req_notify_cq(), the CQ polling/emptying comes after these, so at the time of arm the CQ isn't empty.
