On 3/14/23 10:10 PM, Leon Romanovsky wrote: > On Tue, Mar 14, 2023 at 07:50:19PM +0800, Cheng Xu wrote: >> >> <...> >> >> Our doorbell space is aligned to 4096, this works fine when PAGE_SIZE is >> also 4096, and the doorbell space starts from the mapped page. When >> PAGE_SIZE is not 4096, the doorbell space may starts from the middle of >> the mapped page. >> >> For example, our SQ doorbell starts from the offset 4096 in PCIe bar 0. >> When we map the first SQ doorbell to userspace when PAGE_SIZE is 64K, >> the doorbell space starts from the offset 4096 in mmap returned address. >> >> So the userspace needs to know the doorbell space offset in mmaped page. > > And can't you preserve same alignment in the kernel for doorbells for every page size? > Just always start from 0. > I've considered this option before, but unfortunately can't, at least for CQ DB. The size of our PCIe bar 0 is 512K, and offset [484K, 508K] are CQ doorbells. CQ doorbell space is located in offset [36K, 60K] when PAGE_SIZE = 64K, and can't start from offset 0 in this case. Another reason is that we want to organize SQ doorbell space in unit of 4096. In current implementation, each ucontext will be assigned a SQ doorbell space for both normal doorbell and direct wqe usage. Unit of 4096, compared with larger unit, more ucontexts can be assigned exclusive doorbell space for direct wqe. Thanks, Cheng Xu
