-----"Gal Pressman" <galpress@xxxxxxxxxx> wrote: ----- >To: "Bernard Metzler" <BMT@xxxxxxxxxxxxxx> >From: "Gal Pressman" <galpress@xxxxxxxxxx> >Date: 02/18/2021 01:47PM >Cc: "RDMA mailing list" <linux-rdma@xxxxxxxxxxxxxxx> >Subject: [EXTERNAL] Re: ibv_req_notify_cq clarification > >On 18/02/2021 14:38, Bernard Metzler wrote: >> -----"Gal Pressman" <galpress@xxxxxxxxxx> wrote: ----- >> >>> To: "RDMA mailing list" <linux-rdma@xxxxxxxxxxxxxxx> >>> From: "Gal Pressman" <galpress@xxxxxxxxxx> >>> Date: 02/18/2021 11:26AM >>> Subject: [EXTERNAL] ibv_req_notify_cq clarification >>> >>> I'm a bit confused about the meaning of the ibv_req_notify_cq() >verb: >>> "Upon the addition of a new CQ entry (CQE) to cq, a completion >event >>> will be >>> added to the completion channel associated with the CQ." >>> >>> What is considered a new CQE in this case? >>> The next CQE from the user's perspective, i.e. any new CQE that >>> wasn't consumed >>> by the user's poll cq? >>> Or any new CQE from the device's perspective? >>> >>> For example, if at the time of ibv_req_notify_cq() call the CQ has >>> received 100 >>> completions, but the user hasn't polled his CQ yet, when should he >be >>> notified? >>> On the 101 completion or immediately (since there are completions >>> waiting on the >>> CQ)? >> >> It is from drivers perspective. So notification when 101 >> became available and CQ is marked for notification. >> Applications tend to poll the CQ after requesting >> notification, since there might be a race >> from appl perspective... > >Thanks Bernard, > >Looking at the implementation of rdma-core providers, most of them >pass the CQ >consumer index as part of the arm doorbell which made me think it >arms the CQ to >notify on a completion newer than what the app polled, not on a >completion newer >than the producer index. >Any idea why the consumer index is needed if that's not the case? > ...maybe to do what you suggest - just kicking the completion handler if it appears the application has missed CQE's when the CQ was not armed? That would be in the gray zone of having a provider which wants to be smarter than the application, since this feature is not part of the verbs specification. I'd suggest applications better to rely on the IB_CQ_REPORT_MISSED_EVENTS flag if it doesn't want to pull unconditionally.
