On Wed, Mar 15, 2023 at 09:58:06AM +0800, Cheng Xu wrote: > > > On 3/14/23 10:10 PM, Leon Romanovsky wrote: > > On Tue, Mar 14, 2023 at 07:50:19PM +0800, Cheng Xu wrote: > >> > >> > <...> > >> > >> Our doorbell space is aligned to 4096, this works fine when PAGE_SIZE is > >> also 4096, and the doorbell space starts from the mapped page. When > >> PAGE_SIZE is not 4096, the doorbell space may starts from the middle of > >> the mapped page. > >> > >> For example, our SQ doorbell starts from the offset 4096 in PCIe bar 0. > >> When we map the first SQ doorbell to userspace when PAGE_SIZE is 64K, > >> the doorbell space starts from the offset 4096 in mmap returned address. > >> > >> So the userspace needs to know the doorbell space offset in mmaped page. > > > > And can't you preserve same alignment in the kernel for doorbells for every page size? > > Just always start from 0. > > > > I've considered this option before, but unfortunately can't, at least for CQ DB. > The size of our PCIe bar 0 is 512K, and offset [484K, 508K] are CQ doorbells. > CQ doorbell space is located in offset [36K, 60K] when PAGE_SIZE = 64K, and can't > start from offset 0 in this case. > > Another reason is that we want to organize SQ doorbell space in unit of 4096. > In current implementation, each ucontext will be assigned a SQ doorbell space > for both normal doorbell and direct wqe usage. Unit of 4096, compared with > larger unit, more ucontexts can be assigned exclusive doorbell space for direct > wqe. I have a feeling that there is an existing API for it already. Let's give a chance for Jason to chime in. Thanks > > Thanks, > Cheng Xu
