-----"Jason Gunthorpe" <jgg@xxxxxxxx> wrote: ----- >To: "Gal Pressman" <galpress@xxxxxxxxxx> >From: "Jason Gunthorpe" <jgg@xxxxxxxx> >Date: 02/18/2021 07:35PM >Cc: "RDMA mailing list" <linux-rdma@xxxxxxxxxxxxxxx> >Subject: [EXTERNAL] Re: ibv_req_notify_cq clarification > >On Thu, Feb 18, 2021 at 05:52:16PM +0200, Gal Pressman wrote: >> On 18/02/2021 14:53, Jason Gunthorpe wrote: >> > On Thu, Feb 18, 2021 at 11:13:43AM +0200, Gal Pressman wrote: >> >> I'm a bit confused about the meaning of the ibv_req_notify_cq() >verb: >> >> "Upon the addition of a new CQ entry (CQE) to cq, a completion >event will be >> >> added to the completion channel associated with the CQ." >> >> >> >> What is considered a new CQE in this case? >> >> The next CQE from the user's perspective, i.e. any new CQE that >wasn't consumed >> >> by the user's poll cq? >> >> Or any new CQE from the device's perspective? >> > >> > new CQE from the device perspective. >> > >> >> For example, if at the time of ibv_req_notify_cq() call the CQ >has received 100 >> >> completions, but the user hasn't polled his CQ yet, when should >he be notified? >> >> On the 101 completion or immediately (since there are >completions waiting on the >> >> CQ)? >> > >> > 101 completion >> > >> > It is only meaningful to call it when the CQ is empty. >> >> Thanks, so there's an inherent race between the user's CQ poll and >the next arm? > >I think the specs or man pages talk about this, the application has >to >observe empty, do arm, then poll again then sleep on the cq if empty. > I'd love to see the IB_CQ_REPORT_MISSED_EVENTS flag mechanics available for user land verbs. I learned about this potential race the hard way when a well known distributed FS sometimes hung ;) >> Do you know what's the purpose of the consumer index in the arm >doorbell that's >> implemented by many providers? > >The consumer index is needed by HW to prevent CQ overflow, presumably >the drivers push to reduce the cases where the HW has to read it from >PCI > >Jason >
