Re: [PATCH RFC V2 0/6] rxe_map_mr_sg() fix cleanup and refactor

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 




在 2023/11/9 10:24, Zhijian Li (Fujitsu) 写道:

On 06/11/2023 21:58, Zhu Yanjun wrote:
在 2023/11/6 12:07, Zhijian Li (Fujitsu) 写道:

On 03/11/2023 21:00, Zhu Yanjun wrote:
在 2023/11/3 17:55, Li Zhijian 写道:
I don't collect the Reviewed-by to the patch1-2 this time, since i
think we can make it better.

Patch1-2: Fix kernel panic[1] and benifit to make srp work again.
             Almost nothing change from V1.
Patch3-5: cleanups # newly add
Patch6: make RXE support PAGE_SIZE aligned mr # newly add, but not fully tested

My bad arm64 mechine offten hangs when doing blktests even though i use the
default siw driver.

- nvme and ULPs(rtrs, iser) always registers 4K mr still don't supported yet.
Zhijian

Please read carefully the whole discussion about this problem. You will find a lot of valuable suggestions, especially suggestions from Jason.
Okay, i will read it again. If you can tell me which thread, that would be better.


   From the whole discussion, it seems that the root cause is very clear.
We need to fix this prolem. Please do not send this kind of commits again.

Let's think about what's our goal first.

- 1) Fix the panic[1] and only support PAGE_SIZE MR
- 2) support PAGE_SIZE aligned MR
- 3) support any page_size MR.

I'm sorry i'm not familiar with the linux MM subsystem. It seem it's safe/correct to access
address/memory across pages start from the return of kmap_loca_page(page).
In other words, 2) is already native supported, right?
Yes. Please read the comments from Jason, Leon and Bart. They shared a lot of good advice.
I read the whole discussion again, but I believed i still missed a lost.


 From them, we can know the root cause and how to fix this problem.
I don't think i misunderstood the root cause:
RXE splits memory into PAGE_SIZE units in the xarray. As a result, when we extract an address from the xarray,
we should not access address beyond a PAGE_SIZE window.

This is a complicated problem and it is deeply involved with memory management.

A guy who is very familiar with linux MM is to provide a better solution to this problem.

I expect a whole perfect solution to this problem.

Zhu Yanjun


IIUC, then how to fix it?
- I'm not going to "removing page_size set", it's out of this patch scope.
    Feel free to do the cleanup separately.
- I'm not going to fix the NVMe/rtrs etc problems in this patch set when 64K page is enabled.
    But RXE will tell its callers explicitly "RXE don't don't support such page_size"
- I didn't state RXE supports PAGE_SIZE aligned page_size MR before refactoring rxe_map_mr_sg(),
    because I worry about it was not correct to access address beyond the PAGE_SIZE window.

What I should do next?
Just state "RXE support PAGE_SIZE aligned MR" ? Then patches become
RDMA/rxe: RDMA/rxe: don't allow registering !PAGE_SIZE aligned MR
RDMA/rxe: set RXE_PAGE_SIZE_CAP to starting from PAGE_SIZE

Or just keep we have done in the V1

Thanks


Good Luck.

Zhu Yanjun

I get totally confused now.



Zhu Yanjun

[1] https://lore.kernel.org/all/CAHj4cs9XRqE25jyVw9rj9YugffLn5+f=1znaBEnu1usLOciD+g@xxxxxxxxxxxxxx/T/

Li Zhijian (6):
     RDMA/rxe: RDMA/rxe: don't allow registering !PAGE_SIZE mr
     RDMA/rxe: set RXE_PAGE_SIZE_CAP to PAGE_SIZE
     RDMA/rxe: remove unused rxe_mr.page_shift
     RDMA/rxe: Use PAGE_SIZE and PAGE_SHIFT to extract address from
       page_list
     RDMA/rxe: cleanup rxe_mr.{page_size,page_shift}
     RDMA/rxe: Support PAGE_SIZE aligned MR

    drivers/infiniband/sw/rxe/rxe_mr.c    | 80 ++++++++++++++++-----------
    drivers/infiniband/sw/rxe/rxe_param.h |  2 +-
    drivers/infiniband/sw/rxe/rxe_verbs.h |  9 ---
    3 files changed, 48 insertions(+), 43 deletions(-)




[Index of Archives]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Photo]     [Yosemite News]     [Yosemite Photos]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux