On Fri, Oct 27, 2023 at 12:01:47PM +0800, Zhu Yanjun wrote:
>
> 在 2023/10/27 7:23, Jason Gunthorpe 写道:
> > On Thu, Oct 26, 2023 at 08:59:34PM +0800, Zhu Yanjun wrote:
> > > 在 2023/10/26 19:42, Jason Gunthorpe 写道:
> > > > On Thu, Oct 26, 2023 at 09:05:52AM +0000, Zhijian Li (Fujitsu) wrote:
> > > > > The root cause is that
> > > > >
> > > > > rxe:rxe_set_page() gets wrong when mr.page_size != PAGE_SIZE where it only stores the *page to xarray.
> > > > > So the offset will get lost.
> > > > >
> > > > > For example,
> > > > > store process:
> > > > > page_size = 0x1000;
> > > > > PAGE_SIZE = 0x10000;
> > > > > va0 = 0xffff000020651000;
> > > > > page_offset = 0 = va & (page_size - 1);
> > > > > page = va_to_page(va);
> > > > > xa_store(&mr->page_list, mr->nbuf, page, GFP_KERNEL);
> > > > >
> > > > > load_process:
> > > > > page = xa_load(&mr->page_list, index);
> > > > > page_va = kmap_local_page(page) --> it must be a PAGE_SIZE align value, assume it as 0xffff000020650000
> > > > > va1 = page_va + page_offset = 0xffff000020650000 + 0 = 0xffff000020650000;
> > > > >
> > > > > Obviously, *va0 != va1*, page_offset get lost.
> > > > >
> > > > >
> > > > > How to fix:
> > > > > - revert 325a7eb85199 ("RDMA/rxe: Cleanup page variables in rxe_mr.c")
> > > > > - don't allow ulp registering mr.page_size != PAGE_SIZE ?
> > > > Lets do the second one please. Most devices only support PAGE_SIZE anyhow.
> > > Normally page_size is PAGE_SIZE or the size of the whole compound page (in
> > > the latest kernel version, it is the size of folio). When compound page or
> > > folio is taken into account, the page_size is not equal to
> > > PAGE_SIZE.
> > folios are always multiples of PAGE_SIZE. rxe splits everything into
> > PAGE_SIZE units in the xarray.
> >
> > > If the ULP uses the compound page or folio, the similar problem will occur
> > > again.
> > No, it won't. We never store folios in the xarray.
>
> Sure.
>
> I mean, in ULP, if folio is used, the page size is set to multiple
> PAGE_SIZE, but in RXE, the page size is set to PAGE_SIZE.
>
> So the page size in ULP is different with the page size in RXE.
There is no such thing as a "page size" in the ULP. rxe is the thing
that keeps things in PAGE_SIZE units, and it should be fragmenting
whatever the ulp gives into that. The ULP must simply give virtually
contiguous runs of memory that are PAGE_SIZE aligned
Jason
