On 3 Jan 2024, at 13:03, Jeff Layton wrote: > I'm seeing some hangs when testing with TLS in v6.7-rc8. This kernel has > the patch I sent this morning to get rid of nfsd_put, but otherwise is > stock v6.7-rc8: > > [ 2125.174937] run fstests generic/126 at 2024-01-03 09:46:39 > [ 2129.793577] run fstests generic/127 at 2024-01-03 09:46:44 > [ 3199.661565] run fstests generic/128 at 2024-01-03 10:04:33 > [ 3204.502354] run fstests generic/129 at 2024-01-03 10:04:38 > [ 3208.111189] RPC: Could not send backchannel reply error: -110 > [ 3384.487762] INFO: task looptest:92512 blocked for more than 120 seconds. > [ 3384.491396] Not tainted 6.7.0-rc8-g6c5361baaf84 #74 > [ 3384.494103] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 3384.497215] task:looptest state:D stack:0 pid:92512 tgid:92512 ppid:92333 flags:0x00004002 > [ 3384.499471] Call Trace: > [ 3384.500129] <TASK> > [ 3384.500717] __schedule+0x3c4/0xad0 > [ 3384.501652] schedule+0x31/0xd0 > [ 3384.502455] io_schedule+0x42/0x70 > [ 3384.503344] folio_wait_bit_common+0x121/0x330 > [ 3384.504442] ? __pfx_wake_page_function+0x10/0x10 > [ 3384.505624] folio_wait_writeback+0x27/0x80 > [ 3384.506639] __filemap_fdatawait_range+0x79/0xe0 > [ 3384.507343] filemap_write_and_wait_range+0x81/0xb0 > [ 3384.508043] nfs_wb_all+0x21/0x120 [nfs] > [ 3384.508739] nfs4_file_flush+0x6e/0xb0 [nfsv4] > [ 3384.509499] filp_flush+0x30/0x70 > [ 3384.510045] filp_close+0xf/0x30 > [ 3384.510639] put_files_struct+0x78/0xd0 > [ 3384.511222] do_exit+0x345/0xb10 > [ 3384.511764] ? handle_mm_fault+0x9e/0x360 > [ 3384.512364] ? preempt_count_add+0x47/0xa0 > [ 3384.513012] do_group_exit+0x2d/0x80 > [ 3384.513592] __x64_sys_exit_group+0x14/0x20 > [ 3384.514252] do_syscall_64+0x3f/0xf0 > [ 3384.514809] entry_SYSCALL_64_after_hwframe+0x6e/0x76 > [ 3384.515581] RIP: 0033:0x7f564911595d > [ 3384.516186] RSP: 002b:00007ffecd703338 EFLAGS: 00000202 ORIG_RAX: 00000000000000e7 > [ 3384.517306] RAX: ffffffffffffffda RBX: 00007f5649211fa8 RCX: 00007f564911595d > [ 3384.518368] RDX: 00000000000000e7 RSI: ffffffffffffff28 RDI: 0000000000000000 > [ 3384.519473] RBP: 00007ffecd703390 R08: 00007ffecd7032d8 R09: 00007ffecd70323f > [ 3384.520543] R10: 00007ffecd703190 R11: 0000000000000202 R12: 0000000000000001 > [ 3384.521597] R13: 0000000000000000 R14: 0000000000000000 R15: 00007f5649211fc0 > [ 3384.522647] </TASK> > > [root@kdevops-nfs-tls ~]# cat /sys/kernel/debug/sunrpc/rpc_clnt/*/tasks > 6 5285 0 0x0 0x0 0 nfs41_sequence_ops [nfsv4] nfsv4 SEQUENCE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1460 c805 -11 0x5 0x2a2113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1461 c805 -11 0x5 0x2b2113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1462 c805 -11 0x5 0x2c2113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1463 c805 -11 0x5 0x2d2113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1464 c805 -11 0x5 0x302113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1465 c805 -11 0x5 0x2e2113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1466 c805 -11 0x5 0x2f2113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1467 c805 -11 0x5 0x312113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1468 c805 -11 0x5 0x322113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1469 c805 -11 0x5 0x332113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1470 c805 -11 0x5 0x342113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1471 c805 -11 0x5 0x352113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1472 c805 -11 0x5 0x362113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1473 c805 -11 0x5 0x372113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1474 c805 -11 0x5 0x382113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1475 c805 -11 0x5 0x392113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1476 c805 -11 0x5 0x3a2113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1477 c805 -11 0x5 0x3b2113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1478 c805 -11 0x5 0x3c2113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1479 c805 -11 0x5 0x3d2113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1480 c805 -11 0x5 0x3e2113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1481 c805 -11 0x5 0x3f2113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1482 c805 -11 0x5 0x402113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1483 c805 -11 0x5 0x412113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1484 c805 -11 0x5 0x422113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_connect_status [sunrpc] q:xprt_sending > 1485 c005 -11 0x5 0x432113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_transmit_status [sunrpc] q:xprt_sending > 1486 c005 -11 0x5 0x442113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_transmit_status [sunrpc] q:xprt_sending > 1487 c005 -11 0x5 0x452113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_transmit_status [sunrpc] q:xprt_sending > 1488 c005 -11 0x5 0x462113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_transmit_status [sunrpc] q:xprt_sending > 1489 c005 -11 0x5 0x472113be 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:call_transmit_status [sunrpc] q:xprt_sending > 1490 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1491 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1492 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1493 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1494 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1495 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1496 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1497 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1498 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1499 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1500 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1501 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1502 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1503 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1504 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1505 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1506 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1507 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1508 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1509 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1510 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1511 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1512 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1513 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1514 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1515 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1516 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1517 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1518 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1519 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1520 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1521 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1522 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1523 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1524 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1525 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1526 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1527 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1528 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1529 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1530 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1531 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1532 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1533 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1534 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1535 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1536 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1537 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1538 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1539 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1540 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1541 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1542 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1543 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1544 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1545 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1546 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1547 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1548 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1549 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1550 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1551 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1552 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1553 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1554 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1555 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1556 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1557 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1558 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1559 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1560 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1561 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1562 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1563 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1564 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1565 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1566 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1567 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1568 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1569 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1570 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1571 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1572 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1573 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1574 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1575 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1576 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1577 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1578 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1579 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1580 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1581 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1582 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1583 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1584 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1585 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1586 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1587 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1588 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1589 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1590 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1591 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1592 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1593 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1594 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1595 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1596 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1597 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > 1598 c005 0 0x5 0x0 0 nfs_pgio_common_ops [nfs] nfsv4 WRITE a:rpc_prepare_task [sunrpc] q:ForeChannel Slot table > > In the server, I see this in the log around the same time: > > [ 2344.590202] rpc-srv/tcp: nfsd: got error -90 when sending 112 bytes - shutting down socket > [ 2349.182087] rpc-srv/tcp: nfsd: got error -74 when sending 112 bytes - shutting down socket > [ 2352.685424] rpc-srv/tcp: nfsd: got error -90 when sending 112 bytes - shutting down socket > [ 3583.897575] rpc-srv/tcp: nfsd: got error -104 when sending 112 bytes - shutting down socket This looks like it started out as the problem I've been sending patches to fix on 6.7, latest here: https://lore.kernel.org/linux-nfs/e28038fba1243f00b0dd66b7c5296a1e181645ea.1702496910.git.bcodding@xxxxxxxxxx/ .. however whenever I encounter the issue, the client reconnects the transport again - so I think there might be an additional problem here. It would be interesting to turn up some tracepoints to see what the sunrpc scheduler is doing once the problem occurs. The -EAGAIN tasks should be timing out and reconnecting the transport, I think. Ben
