> On Aug 4, 2020, at 9:46 AM, Leon Romanovsky <leon@xxxxxxxxxx> wrote: > > On Tue, Aug 04, 2020 at 09:12:55AM -0400, Chuck Lever wrote: >> >> >>> On Aug 4, 2020, at 9:08 AM, Timo Rothenpieler <timo@xxxxxxxxxxxxxxxx> wrote: >>> >>> On 04.08.2020 14:49, Chuck Lever wrote: >>>> Timo, I tend to think this is not a configuration issue. >>>> Do you know of a known working kernel? >>> >>> This is a brand new system, it's never been running with any kernel older than 5.4, and downgrading it to 4.19 or something else while in operation is unfortunately not easily possible. For a client it would definitely not be out of the question, but the main nfs server I cannot easily downgrade. >>> >>> Also keep in mind that the dmesg spam happens on both server and client simultaneously. >> >> Let's start with the client only, since restarting it seems to clear the problem. > > It is client because according to the server CQE errors, it is Remote_Invalid_Request_Error > with "9.7.5.2.2 NAK CODES" from IBTA. Thanks! OK, then let's use ftrace. Timo, can you install trace-cmd on your client? Then: 1. # trace-cmd record -e rpcrdma -e sunrpc 2. Trigger the problem 3. Control-C the trace-cmd, and copy the trace.dat file to another system 4. reboot your client Then send me your trace.dat. You don't have to cc the mailing lists. -- Chuck Lever
