Hi, We are using 8 virtual machines (some ESXi 6.0, some Proxmox 8.1.3) to execute scripts which read document files and write results in a database. Reading 40k documents in a minute is nothing special. Starting with 2024 we are getting problems and have to schedule weekly reboots. We did the last client upgrades on 30. November 2023 and server upgrades on 23. June 2023. But we created a new ZFS File System with sharenfs for 2024. First error happened on the night between 10. - 11. January 2024. Cpu sys time avg is 5-10% for the first 24h after a reboot and 25-35% for the last 24h before an issue occurres. We don't know how to reproduce the error besides: read much for a long time.. We are used to having 5-20% cpu sys avg in 2023 without issues. We know an issue occurred after: - Nfs client value test_stateid increases over 10k (highest went from 6k to over 4M in 3h) and every nfs request slows down, higher test_stateid means client getting slower. Clients keep being slow even after you kill all processes and start a single performance test. No other clientvalue changes much. - "rpcdebug -m nfs -s all" shows: Jan 15 13:47:29 merkur6 kernel: nfs41_handle_sequence_flag_errors: "192.168.2.233" (client ID 8ed80a651d5a6f4a) flags=0x00000040 All 8 Clients had this Problem once or more times this month. We do not use any security options. Slow Client: root@merkur6:/var/log# time ls -l /dokuserver/anzeigen/2024/01/11/11/ real 1m11.308s user 0m0.239s sys 1m5.375s Fine Client: root@merkur3:~# time ls -l /dokuserver/anzeigen/2024/01/11/11 | wc -l 47117 real 0m2.222s user 0m0.307s sys 0m1.502s Server: root@herodot:~# zfs get sharenfs herodot_zfs_pool/data/dokumente/anzeigen/2024 NAME PROPERTY VALUE SOURCE herodot_zfs_pool/data/dokumente/anzeigen/2024 sharenfs rw,no_root_squash,sync,no_subtree_check,insecure local root@herodot:~# zfs version zfs-2.1.5-1ubuntu6~22.04.2 zfs-kmod-2.1.5-1ubuntu6~22.04.1 root@herodot:~# uname -a Linux herodot 5.15.0-75-generic #82-Ubuntu SMP Tue Jun 6 23:10:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux root@herodot:~# exportfs -v /data/dokumente 192.168.102.110(sync,wdelay,hide,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/dokumente 192.168.102.14(sync,wdelay,hide,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/dokumente 192.168.2.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/dokumente 192.168.3.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/dokumente 172.16.2.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash) /data/dokumente 10.8.0.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/dokumente 192.168.103.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/dokumente 10.8.0.0/23(sync,wdelay,hide,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/scanner 192.168.2.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/scanner 192.168.3.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/scanner 10.8.0.0/23(sync,wdelay,hide,no_subtree_check,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data <world>(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/dokumente <world>(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/dokumente/anzeigen <world>(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/dokumente/anzeigen/2021 <world>(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/dokumente/anzeigen/2022 <world>(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/dokumente/anzeigen/2023 <world>(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/dokumente/anzeigen/FIRMENLOGOS <world>(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,insecure,no_root_squash,no_all_squash) /herodot_zfs_pool <world>(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,insecure,no_root_squash,no_all_squash) /data/dokumente/anzeigen/2024 <world>(sync,wdelay,hide,no_subtree_check,mountpoint,sec=sys,rw,insecure,no_root_squash,no_all_squash) First Client (merkur7 /Proxmox Client): root@merkur7:~# uname -a Linux merkur7 6.5.0-14-generic #14~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 20 18:15:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux root@merkur7:~# mount -l [removed local mounts..] 192.168.2.240:/exporte on /exporte type nfs4 (rw,relatime,vers=4.2,rsize=524288,wsize=524288,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.240) 192.168.2.233:/data/dokumente on /dokuserver type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.233) 192.168.2.233:/data/dokumente/anzeigen on /dokuserver/anzeigen type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.233) 192.168.2.168:/data/dokumente/anzeigen/2005 on /dokuserver/anzeigen/2005 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2006 on /dokuserver/anzeigen/2006 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2003 on /dokuserver/anzeigen/2003 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2008 on /dokuserver/anzeigen/2008 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2010 on /dokuserver/anzeigen/2010 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2014 on /dokuserver/anzeigen/2014 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2011 on /dokuserver/anzeigen/2011 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2004 on /dokuserver/anzeigen/2004 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2012 on /dokuserver/anzeigen/2012 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2019 on /dokuserver/anzeigen/2019 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2007 on /dokuserver/anzeigen/2007 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2015 on /dokuserver/anzeigen/2015 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2017 on /dokuserver/anzeigen/2017 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2016 on /dokuserver/anzeigen/2016 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2020 on /dokuserver/anzeigen/2020 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2009 on /dokuserver/anzeigen/2009 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2018 on /dokuserver/anzeigen/2018 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2013 on /dokuserver/anzeigen/2013 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.191,local_lock=none,addr=192.168.2.168) Second Client (merkur6 /ESXi Client): root@merkur6:~# uname -a Linux merkur6 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux root@merkur6:~# mount -l [removed local mounts..] 192.168.2.233:/data/dokumente on /dokuserver type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.233) 192.168.2.240:/exporte on /exporte type nfs4 (rw,relatime,vers=4.2,rsize=524288,wsize=524288,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.240) 192.168.2.233:/data/dokumente/anzeigen on /dokuserver/anzeigen type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.233) 192.168.2.168:/data/dokumente/anzeigen/2010 on /dokuserver/anzeigen/2010 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2008 on /dokuserver/anzeigen/2008 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2007 on /dokuserver/anzeigen/2007 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2004 on /dokuserver/anzeigen/2004 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2019 on /dokuserver/anzeigen/2019 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2003 on /dokuserver/anzeigen/2003 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2016 on /dokuserver/anzeigen/2016 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2005 on /dokuserver/anzeigen/2005 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2020 on /dokuserver/anzeigen/2020 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2006 on /dokuserver/anzeigen/2006 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2009 on /dokuserver/anzeigen/2009 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2014 on /dokuserver/anzeigen/2014 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2015 on /dokuserver/anzeigen/2015 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2011 on /dokuserver/anzeigen/2011 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2013 on /dokuserver/anzeigen/2013 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2012 on /dokuserver/anzeigen/2012 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2018 on /dokuserver/anzeigen/2018 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.168:/data/dokumente/anzeigen/2017 on /dokuserver/anzeigen/2017 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.168) 192.168.2.233:/data/dokumente/anzeigen/2024 on /dokuserver/anzeigen/2024 type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.2.184,local_lock=none,addr=192.168.2.233) Some /var/log/syslog after an issue occurred: Jan 15 13:46:56 merkur6 kernel: nfs4_renew_state: start Jan 15 13:46:56 merkur6 kernel: nfs4_schedule_state_renewal: requeueing work. Lease period = 33 Jan 15 13:46:56 merkur6 kernel: nfs4_renew_state: done Jan 15 13:46:59 merkur6 CRON[3797003]: pam_unix(cron:session): session closed for user root Jan 15 13:47:00 merkur6 kernel: nfs4_renew_state: start Jan 15 13:47:00 merkur6 kernel: <-- nfs41_proc_async_sequence status=0 Jan 15 13:47:00 merkur6 kernel: nfs4_renew_state: done Jan 15 13:47:00 merkur6 kernel: --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=30 Jan 15 13:47:00 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0 Jan 15 13:47:00 merkur6 kernel: encode_sequence: sessionid=1701864457:3198674332:283960:0 seqid=278 slotid=0 max_slotid=0 cache_this=0 Jan 15 13:47:00 merkur6 kernel: --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=30 Jan 15 13:47:00 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1 Jan 15 13:47:00 merkur6 kernel: nfs4_free_slot: slotid 1 highest_used_slotid 0 Jan 15 13:47:00 merkur6 kernel: nfs41_sequence_process: Error 0 free the slot Jan 15 13:47:00 merkur6 kernel: nfs4_free_slot: slotid 0 highest_used_slotid 4294967295 Jan 15 13:47:00 merkur6 kernel: nfs41_sequence_call_done rpc_cred 000000006b5e23dc Jan 15 13:47:00 merkur6 kernel: <-- nfs41_sequence_call_done Jan 15 13:47:00 merkur6 kernel: nfs4_schedule_state_renewal: requeueing work. Lease period = 60 Jan 15 13:47:01 merkur6 CRON[3797168]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0) Jan 15 13:47:01 merkur6 CRON[3797169]: (root) CMD (/usr/libexec/zabbix-extensions/scripts/iostat-collect.sh /tmp/iostat-cron.out 59 >/dev/null 2>&1) Jan 15 13:47:04 merkur6 rsyslogd[3776903]: cannot connect to 10.255.1.14:514: Connection timed out [v8.2112.0 try https://www.rsyslog.com/e/2027 ] Jan 15 13:47:04 merkur6 kernel: nfs4_renew_state: start Jan 15 13:47:04 merkur6 kernel: <-- nfs41_proc_async_sequence status=0 Jan 15 13:47:04 merkur6 kernel: --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=30 Jan 15 13:47:04 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0 Jan 15 13:47:04 merkur6 kernel: encode_sequence: sessionid=1686681223:1972710396:429668:0 seqid=261856 slotid=0 max_slotid=0 cache_this=0 Jan 15 13:47:04 merkur6 kernel: nfs4_renew_state: done Jan 15 13:47:04 merkur6 kernel: --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=30 Jan 15 13:47:04 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1 Jan 15 13:47:04 merkur6 kernel: nfs4_free_slot: slotid 1 highest_used_slotid 0 Jan 15 13:47:04 merkur6 kernel: nfs41_sequence_process: Error 0 free the slot Jan 15 13:47:04 merkur6 kernel: nfs4_free_slot: slotid 0 highest_used_slotid 4294967295 Jan 15 13:47:04 merkur6 kernel: nfs41_sequence_call_done rpc_cred 000000006b5e23dc Jan 15 13:47:04 merkur6 kernel: <-- nfs41_sequence_call_done Jan 15 13:47:04 merkur6 kernel: nfs4_schedule_state_renewal: requeueing work. Lease period = 60 Jan 15 13:47:29 merkur6 kernel: NFS: permission(0:45/34), mask=0x81, res=-10 Jan 15 13:47:29 merkur6 kernel: NFS: revalidating (0:45/34) Jan 15 13:47:29 merkur6 kernel: --> nfs41_call_sync_prepare data->seq_server 00000000626f52bd Jan 15 13:47:29 merkur6 kernel: --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=30 Jan 15 13:47:29 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0 Jan 15 13:47:29 merkur6 kernel: encode_sequence: sessionid=1695209614:1248811549:744489:0 seqid=74093603 slotid=0 max_slotid=0 cache_this=0 Jan 15 13:47:29 merkur6 kernel: decode_attr_type: type=040000 Jan 15 13:47:29 merkur6 kernel: decode_attr_change: change attribute=7287863876211365669 Jan 15 13:47:29 merkur6 kernel: decode_attr_size: file size=143 Jan 15 13:47:29 merkur6 kernel: decode_attr_fsid: fsid=(0xd9e5dc8000a3ddde/0x0) Jan 15 13:47:29 merkur6 kernel: decode_attr_fileid: fileid=34 Jan 15 13:47:29 merkur6 kernel: decode_attr_fs_locations: fs_locations done, error = 0 Jan 15 13:47:29 merkur6 kernel: decode_attr_mode: file mode=0775 Jan 15 13:47:29 merkur6 kernel: decode_attr_nlink: nlink=72 Jan 15 13:47:29 merkur6 kernel: decode_attr_owner: uid=65534 Jan 15 13:47:29 merkur6 kernel: decode_attr_group: gid=65534 Jan 15 13:47:29 merkur6 kernel: decode_attr_rdev: rdev=(0x0:0x0) Jan 15 13:47:29 merkur6 kernel: decode_attr_space_used: space used=81408 Jan 15 13:47:29 merkur6 kernel: decode_attr_time_access: atime=1686643653 Jan 15 13:47:29 merkur6 kernel: decode_attr_time_metadata: ctime=1696838037 Jan 15 13:47:29 merkur6 kernel: decode_attr_time_modify: mtime=1696838037 Jan 15 13:47:29 merkur6 kernel: decode_attr_mounted_on_fileid: fileid=2 Jan 15 13:47:29 merkur6 kernel: decode_getfattr_attrs: xdr returned 0 Jan 15 13:47:29 merkur6 kernel: decode_getfattr_generic: xdr returned 0 Jan 15 13:47:29 merkur6 kernel: nfs41_handle_sequence_flag_errors: "192.168.2.233" (client ID 8ed80a651d5a6f4a) flags=0x00000040 Jan 15 13:47:29 merkur6 kernel: nfs41_handle_recallable_state_revoked: Recallable state revoked on server 192.168.2.233! Jan 15 13:47:29 merkur6 kernel: --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=30 Jan 15 13:47:29 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1 Jan 15 13:47:29 merkur6 kernel: nfs4_free_slot: slotid 1 highest_used_slotid 0 Jan 15 13:47:29 merkur6 kernel: nfs41_sequence_process: Error 0 free the slot Jan 15 13:47:29 merkur6 kernel: nfs4_free_slot: slotid 0 highest_used_slotid 4294967295 Jan 15 13:47:29 merkur6 kernel: NFS: nfs_update_inode(0:45/34 fh_crc=0x227dee8d ct=2 info=0x427e7f) Jan 15 13:47:29 merkur6 kernel: NFS: (0:45/34) revalidation complete Jan 15 13:47:29 merkur6 kernel: NFS: permission(0:45/34), mask=0x1, res=0 Jan 15 13:47:29 merkur6 kernel: NFS: nfs_lookup_revalidate_done(/anzeigen) is valid Jan 15 13:47:29 merkur6 kernel: NFS: revalidating (0:43/34) Jan 15 13:47:29 merkur6 kernel: --> nfs41_call_sync_prepare data->seq_server 00000000c979f144 Jan 15 13:47:29 merkur6 kernel: --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=30 Jan 15 13:47:29 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0 Jan 15 13:47:29 merkur6 kernel: encode_sequence: sessionid=1695209614:1248811549:744489:0 seqid=74093604 slotid=0 max_slotid=0 cache_this=0 Jan 15 13:47:29 merkur6 kernel: decode_attr_type: type=040000 Jan 15 13:47:29 merkur6 kernel: decode_attr_change: change attribute=7307486567865913980 Jan 15 13:47:29 merkur6 kernel: decode_attr_size: file size=43 Jan 15 13:47:29 merkur6 kernel: decode_attr_fsid: fsid=(0xd0878adf00f27b7f/0x0) Jan 15 13:47:29 merkur6 kernel: decode_attr_fileid: fileid=34 Jan 15 13:47:29 merkur6 kernel: decode_attr_fs_locations: fs_locations done, error = 0 Jan 15 13:47:29 merkur6 kernel: decode_attr_mode: file mode=0775 Jan 15 13:47:29 merkur6 kernel: decode_attr_nlink: nlink=42 Jan 15 13:47:29 merkur6 kernel: decode_attr_owner: uid=65534 ...skipping... Jan 15 13:47:29 merkur6 kernel: NFS: nfs_lookup_revalidate_done(/-) is valid Jan 15 13:47:29 merkur6 kernel: NFS: dentry_delete(/-, 8084c) Jan 15 13:47:29 merkur6 kernel: NFS: permission(0:45/34), mask=0x81, res=0 Jan 15 13:47:29 merkur6 kernel: NFS: nfs_lookup_revalidate_done(/anzeigen) is valid Jan 15 13:47:29 merkur6 kernel: NFS: permission(0:43/34), mask=0x81, res=0 Jan 15 13:47:29 merkur6 kernel: NFS: nfs_lookup_revalidate_done(/2024) is valid Jan 15 13:47:29 merkur6 kernel: NFS: permission(0:75/34), mask=0x81, res=0 Jan 15 13:47:29 merkur6 kernel: NFS: nfs_lookup_revalidate_done(/-) is valid Jan 15 13:47:31 merkur6 kernel: nfs4_renew_state: start Jan 15 13:47:31 merkur6 kernel: nfs4_schedule_state_renewal: requeueing work. Lease period = 59 Jan 15 13:47:31 merkur6 kernel: nfs4_renew_state: done Jan 15 13:48:00 merkur6 CRON[3797168]: pam_unix(cron:session): session closed for user root Jan 15 13:48:00 merkur6 CRON[3797195]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0) Jan 15 13:48:00 merkur6 CRON[3797196]: (root) CMD (/usr/libexec/zabbix-extensions/scripts/iostat-collect.sh /tmp/iostat-cron.out 59 >/dev/null 2>&1) Jan 15 13:48:01 merkur6 kernel: nfs4_renew_state: start Jan 15 13:48:01 merkur6 kernel: <-- nfs41_proc_async_sequence status=0 Jan 15 13:48:01 merkur6 kernel: nfs4_renew_state: done Jan 15 13:48:01 merkur6 kernel: --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=30 Jan 15 13:48:01 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0 Jan 15 13:48:01 merkur6 kernel: encode_sequence: sessionid=1701864457:3198674332:283960:0 seqid=279 slotid=0 max_slotid=0 cache_this=0 Jan 15 13:48:01 merkur6 kernel: --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=30 Jan 15 13:48:01 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1 Jan 15 13:48:01 merkur6 kernel: nfs4_free_slot: slotid 1 highest_used_slotid 0 Jan 15 13:48:01 merkur6 kernel: nfs41_sequence_process: Error 0 free the slot Jan 15 13:48:01 merkur6 kernel: nfs4_free_slot: slotid 0 highest_used_slotid 4294967295 Jan 15 13:48:01 merkur6 kernel: nfs41_sequence_call_done rpc_cred 000000006b5e23dc Jan 15 13:48:01 merkur6 kernel: <-- nfs41_sequence_call_done Jan 15 13:48:01 merkur6 kernel: nfs4_schedule_state_renewal: requeueing work. Lease period = 60 Jan 15 13:48:06 merkur6 kernel: nfs4_renew_state: start Jan 15 13:48:06 merkur6 kernel: <-- nfs41_proc_async_sequence status=0 Jan 15 13:48:06 merkur6 kernel: --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=30 Jan 15 13:48:06 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0 Jan 15 13:48:06 merkur6 kernel: encode_sequence: sessionid=1686681223:1972710396:429668:0 seqid=261857 slotid=0 max_slotid=0 cache_this=0 Jan 15 13:48:06 merkur6 kernel: nfs4_renew_state: done Jan 15 13:48:06 merkur6 kernel: --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=30 Jan 15 13:48:06 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1 Jan 15 13:48:06 merkur6 kernel: nfs4_free_slot: slotid 1 highest_used_slotid 0 Jan 15 13:48:06 merkur6 kernel: nfs41_sequence_process: Error 0 free the slot Jan 15 13:48:06 merkur6 kernel: nfs4_free_slot: slotid 0 highest_used_slotid 4294967295 Jan 15 13:48:06 merkur6 kernel: nfs41_sequence_call_done rpc_cred 000000006b5e23dc Jan 15 13:48:06 merkur6 kernel: <-- nfs41_sequence_call_done Jan 15 13:48:06 merkur6 kernel: nfs4_schedule_state_renewal: requeueing work. Lease period = 60 ...skipping... Jan 15 13:47:29 merkur6 kernel: NFS: nfs_lookup_revalidate_done(/-) is valid Jan 15 13:47:29 merkur6 kernel: NFS: dentry_delete(/-, 8084c) Jan 15 13:47:29 merkur6 kernel: NFS: permission(0:45/34), mask=0x81, res=0 Jan 15 13:47:29 merkur6 kernel: NFS: nfs_lookup_revalidate_done(/anzeigen) is valid Jan 15 13:47:29 merkur6 kernel: NFS: permission(0:43/34), mask=0x81, res=0 Jan 15 13:47:29 merkur6 kernel: NFS: nfs_lookup_revalidate_done(/2024) is valid Jan 15 13:47:29 merkur6 kernel: NFS: permission(0:75/34), mask=0x81, res=0 Jan 15 13:47:29 merkur6 kernel: NFS: nfs_lookup_revalidate_done(/-) is valid Jan 15 13:47:31 merkur6 kernel: nfs4_renew_state: start Jan 15 13:47:31 merkur6 kernel: nfs4_schedule_state_renewal: requeueing work. Lease period = 59 Jan 15 13:47:31 merkur6 kernel: nfs4_renew_state: done Jan 15 13:48:00 merkur6 CRON[3797168]: pam_unix(cron:session): session closed for user root Jan 15 13:48:00 merkur6 CRON[3797195]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0) Jan 15 13:48:00 merkur6 CRON[3797196]: (root) CMD (/usr/libexec/zabbix-extensions/scripts/iostat-collect.sh /tmp/iostat-cron.out 59 >/dev/null 2>&1) Jan 15 13:48:01 merkur6 kernel: nfs4_renew_state: start Jan 15 13:48:01 merkur6 kernel: <-- nfs41_proc_async_sequence status=0 Jan 15 13:48:01 merkur6 kernel: nfs4_renew_state: done Jan 15 13:48:01 merkur6 kernel: --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=30 Jan 15 13:48:01 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0 Jan 15 13:48:01 merkur6 kernel: encode_sequence: sessionid=1701864457:3198674332:283960:0 seqid=279 slotid=0 max_slotid=0 cache_this=0 Jan 15 13:48:01 merkur6 kernel: --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=30 Jan 15 13:48:01 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1 Jan 15 13:48:01 merkur6 kernel: nfs4_free_slot: slotid 1 highest_used_slotid 0 Jan 15 13:48:01 merkur6 kernel: nfs41_sequence_process: Error 0 free the slot Jan 15 13:48:01 merkur6 kernel: nfs4_free_slot: slotid 0 highest_used_slotid 4294967295 Jan 15 13:48:01 merkur6 kernel: nfs41_sequence_call_done rpc_cred 000000006b5e23dc Jan 15 13:48:01 merkur6 kernel: <-- nfs41_sequence_call_done Jan 15 13:48:01 merkur6 kernel: nfs4_schedule_state_renewal: requeueing work. Lease period = 60 Jan 15 13:48:06 merkur6 kernel: nfs4_renew_state: start Jan 15 13:48:06 merkur6 kernel: <-- nfs41_proc_async_sequence status=0 Jan 15 13:48:06 merkur6 kernel: --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=30 Jan 15 13:48:06 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0 Jan 15 13:48:06 merkur6 kernel: encode_sequence: sessionid=1686681223:1972710396:429668:0 seqid=261857 slotid=0 max_slotid=0 cache_this=0 Jan 15 13:48:06 merkur6 kernel: nfs4_renew_state: done Jan 15 13:48:06 merkur6 kernel: --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=30 Jan 15 13:48:06 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1 Jan 15 13:48:06 merkur6 kernel: nfs4_free_slot: slotid 1 highest_used_slotid 0 Jan 15 13:48:06 merkur6 kernel: nfs41_sequence_process: Error 0 free the slot Jan 15 13:48:06 merkur6 kernel: nfs4_free_slot: slotid 0 highest_used_slotid 4294967295 Jan 15 13:48:06 merkur6 kernel: nfs41_sequence_call_done rpc_cred 000000006b5e23dc Jan 15 13:48:06 merkur6 kernel: <-- nfs41_sequence_call_done Jan 15 13:48:06 merkur6 kernel: nfs4_schedule_state_renewal: requeueing work. Lease period = 60 ...skipping... Jan 15 13:47:29 merkur6 kernel: NFS: nfs_lookup_revalidate_done(/-) is valid Jan 15 13:47:29 merkur6 kernel: NFS: dentry_delete(/-, 8084c) Jan 15 13:47:29 merkur6 kernel: NFS: permission(0:45/34), mask=0x81, res=0 Jan 15 13:47:29 merkur6 kernel: NFS: nfs_lookup_revalidate_done(/anzeigen) is valid Jan 15 13:47:29 merkur6 kernel: NFS: permission(0:43/34), mask=0x81, res=0 Jan 15 13:47:29 merkur6 kernel: NFS: nfs_lookup_revalidate_done(/2024) is valid Jan 15 13:47:29 merkur6 kernel: NFS: permission(0:75/34), mask=0x81, res=0 Jan 15 13:47:29 merkur6 kernel: NFS: nfs_lookup_revalidate_done(/-) is valid Jan 15 13:47:31 merkur6 kernel: nfs4_renew_state: start Jan 15 13:47:31 merkur6 kernel: nfs4_schedule_state_renewal: requeueing work. Lease period = 59 Jan 15 13:47:31 merkur6 kernel: nfs4_renew_state: done Jan 15 13:48:00 merkur6 CRON[3797168]: pam_unix(cron:session): session closed for user root Jan 15 13:48:00 merkur6 CRON[3797195]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0) Jan 15 13:48:00 merkur6 CRON[3797196]: (root) CMD (/usr/libexec/zabbix-extensions/scripts/iostat-collect.sh /tmp/iostat-cron.out 59 >/dev/null 2>&1) Jan 15 13:48:01 merkur6 kernel: nfs4_renew_state: start Jan 15 13:48:01 merkur6 kernel: <-- nfs41_proc_async_sequence status=0 Jan 15 13:48:01 merkur6 kernel: nfs4_renew_state: done Jan 15 13:48:01 merkur6 kernel: --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=30 Jan 15 13:48:01 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0 Jan 15 13:48:01 merkur6 kernel: encode_sequence: sessionid=1701864457:3198674332:283960:0 seqid=279 slotid=0 max_slotid=0 cache_this=0 Jan 15 13:48:01 merkur6 kernel: --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=30 Jan 15 13:48:01 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1 Jan 15 13:48:01 merkur6 kernel: nfs4_free_slot: slotid 1 highest_used_slotid 0 Jan 15 13:48:01 merkur6 kernel: nfs41_sequence_process: Error 0 free the slot Jan 15 13:48:01 merkur6 kernel: nfs4_free_slot: slotid 0 highest_used_slotid 4294967295 Jan 15 13:48:01 merkur6 kernel: nfs41_sequence_call_done rpc_cred 000000006b5e23dc Jan 15 13:48:01 merkur6 kernel: <-- nfs41_sequence_call_done Jan 15 13:48:01 merkur6 kernel: nfs4_schedule_state_renewal: requeueing work. Lease period = 60 Jan 15 13:48:06 merkur6 kernel: nfs4_renew_state: start Jan 15 13:48:06 merkur6 kernel: <-- nfs41_proc_async_sequence status=0 Jan 15 13:48:06 merkur6 kernel: --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=30 Jan 15 13:48:06 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0 Jan 15 13:48:06 merkur6 kernel: encode_sequence: sessionid=1686681223:1972710396:429668:0 seqid=261857 slotid=0 max_slotid=0 cache_this=0 Jan 15 13:48:06 merkur6 kernel: nfs4_renew_state: done Jan 15 13:48:06 merkur6 kernel: --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=30 Jan 15 13:48:06 merkur6 kernel: <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1 Jan 15 13:48:06 merkur6 kernel: nfs4_free_slot: slotid 1 highest_used_slotid 0 Jan 15 13:48:06 merkur6 kernel: nfs41_sequence_process: Error 0 free the slot Jan 15 13:48:06 merkur6 kernel: nfs4_free_slot: slotid 0 highest_used_slotid 4294967295 Jan 15 13:48:06 merkur6 kernel: nfs41_sequence_call_done rpc_cred 000000006b5e23dc Jan 15 13:48:06 merkur6 kernel: <-- nfs41_sequence_call_done Jan 15 13:48:06 merkur6 kernel: nfs4_schedule_state_renewal: requeueing work. Lease period = 60 I hope this bug report is okay. Regards Sebastian
