在 2023/11/9 10:24, Zhijian Li (Fujitsu) 写道:
On 06/11/2023 21:58, Zhu Yanjun wrote:
在 2023/11/6 12:07, Zhijian Li (Fujitsu) 写道:
On 03/11/2023 21:00, Zhu Yanjun wrote:
在 2023/11/3 17:55, Li Zhijian 写道:
I don't collect the Reviewed-by to the patch1-2 this time, since i
think we can make it better.
Patch1-2: Fix kernel panic[1] and benifit to make srp work again.
Almost nothing change from V1.
Patch3-5: cleanups # newly add
Patch6: make RXE support PAGE_SIZE aligned mr # newly add, but not fully tested
My bad arm64 mechine offten hangs when doing blktests even though i use the
default siw driver.
- nvme and ULPs(rtrs, iser) always registers 4K mr still don't supported yet.
Zhijian
Please read carefully the whole discussion about this problem. You will find a lot of valuable suggestions, especially suggestions from Jason.
Okay, i will read it again. If you can tell me which thread, that would be better.
From the whole discussion, it seems that the root cause is very clear.
We need to fix this prolem. Please do not send this kind of commits again.
Let's think about what's our goal first.
- 1) Fix the panic[1] and only support PAGE_SIZE MR
- 2) support PAGE_SIZE aligned MR
- 3) support any page_size MR.
I'm sorry i'm not familiar with the linux MM subsystem. It seem it's safe/correct to access
address/memory across pages start from the return of kmap_loca_page(page).
In other words, 2) is already native supported, right?
Yes. Please read the comments from Jason, Leon and Bart. They shared a lot of good advice.
I read the whole discussion again, but I believed i still missed a lost.
From them, we can know the root cause and how to fix this problem.
I don't think i misunderstood the root cause:
RXE splits memory into PAGE_SIZE units in the xarray. As a result, when we extract an address from the xarray,
we should not access address beyond a PAGE_SIZE window.
This is a complicated problem and it is deeply involved with memory
management.
A guy who is very familiar with linux MM is to provide a better solution
to this problem.
I expect a whole perfect solution to this problem.
Zhu Yanjun
IIUC, then how to fix it?
- I'm not going to "removing page_size set", it's out of this patch scope.
Feel free to do the cleanup separately.
- I'm not going to fix the NVMe/rtrs etc problems in this patch set when 64K page is enabled.
But RXE will tell its callers explicitly "RXE don't don't support such page_size"
- I didn't state RXE supports PAGE_SIZE aligned page_size MR before refactoring rxe_map_mr_sg(),
because I worry about it was not correct to access address beyond the PAGE_SIZE window.
What I should do next?
Just state "RXE support PAGE_SIZE aligned MR" ? Then patches become
RDMA/rxe: RDMA/rxe: don't allow registering !PAGE_SIZE aligned MR
RDMA/rxe: set RXE_PAGE_SIZE_CAP to starting from PAGE_SIZE
Or just keep we have done in the V1
Thanks
Good Luck.
Zhu Yanjun
I get totally confused now.
Zhu Yanjun
[1] https://lore.kernel.org/all/CAHj4cs9XRqE25jyVw9rj9YugffLn5+f=1znaBEnu1usLOciD+g@xxxxxxxxxxxxxx/T/
Li Zhijian (6):
RDMA/rxe: RDMA/rxe: don't allow registering !PAGE_SIZE mr
RDMA/rxe: set RXE_PAGE_SIZE_CAP to PAGE_SIZE
RDMA/rxe: remove unused rxe_mr.page_shift
RDMA/rxe: Use PAGE_SIZE and PAGE_SHIFT to extract address from
page_list
RDMA/rxe: cleanup rxe_mr.{page_size,page_shift}
RDMA/rxe: Support PAGE_SIZE aligned MR
drivers/infiniband/sw/rxe/rxe_mr.c | 80 ++++++++++++++++-----------
drivers/infiniband/sw/rxe/rxe_param.h | 2 +-
drivers/infiniband/sw/rxe/rxe_verbs.h | 9 ---
3 files changed, 48 insertions(+), 43 deletions(-)
