On Wed, Mar 16, 2022 at 11:16 PM Sagi Grimberg <sagi@xxxxxxxxxxx> wrote: > > > >> Hi Yi Zhang, > >> > >> thanks for testing the patches. > >> > >> Can you provide more info on the time it took with both kernels ? > > > > Hi Max > > Sorry for the late response, here are the test results/dmesg on > > debug/non-debug kernel with your patch: > > debug kernel: timeout > > # time nvme connect -t rdma -a 172.31.0.202 -s 4420 -n testnqn > > real 0m16.956s > > user 0m0.000s > > sys 0m0.237s > > # time nvme reset /dev/nvme0 > > real 1m33.623s > > user 0m0.000s > > sys 0m0.024s > > # time nvme disconnect-all > > real 1m26.640s > > user 0m0.000s > > sys 0m9.969s > > > > host dmesg: > > https://pastebin.com/8T3Lqtkn > > target dmesg: > > https://pastebin.com/KpFP7xG2 > > > > non-debug kernel: no timeout issue, but still 12s for reset, and 8s > > for disconnect > > host: > > # time nvme connect -t rdma -a 172.31.0.202 -s 4420 -n testnqn > > > > real 0m4.579s > > user 0m0.000s > > sys 0m0.004s > > # time nvme reset /dev/nvme0 > > > > real 0m12.778s > > user 0m0.000s > > sys 0m0.006s > > # time nvme reset /dev/nvme0 > > > > real 0m12.793s > > user 0m0.000s > > sys 0m0.006s > > # time nvme reset /dev/nvme0 > > > > real 0m12.808s > > user 0m0.000s > > sys 0m0.006s > > # time nvme disconnect-all > > > > real 0m8.348s > > user 0m0.000s > > sys 0m0.189s > > These are very long times for a non-debug kernel... > Max, do you see the root cause for this? > > Yi, does this happen with rxe/siw as well? Hi Sagi rxe/siw will take less than 1s with rdma_rxe # time nvme reset /dev/nvme0 real 0m0.094s user 0m0.000s sys 0m0.006s with siw # time nvme reset /dev/nvme0 real 0m0.097s user 0m0.000s sys 0m0.006s This is only reproducible with mlx IB card, as I mentioned before, the reset operation time changed from 3s to 12s after the below commit, could you check this commit? commit 5ec5d3bddc6b912b7de9e3eb6c1f2397faeca2bc Author: Max Gurtovoy <maxg@xxxxxxxxxxxx> Date: Tue May 19 17:05:56 2020 +0300 nvme-rdma: add metadata/T10-PI support > -- Best Regards, Yi Zhang
