Have you checked inode usage (df -i /lvbackups/brick ) ?
Best Regards,
Strahil Nikolov
On Tuesday, May 2, 2023, 3:05 AM, brandon@xxxxxxxxxxxxx wrote:
Hi Gluster users,We are seeing 'error=No space left on device' issue and hoping someone mightcould advise?We are using a 12 node glusterfs v10.4 distributed vsftpd backup cluster foryears (not new) and recently 2 weeks ago upgraded to v9 > v10.4. I do notknow if the upgrade is related to this new issue.We are seeing a new issue 'error=No space left on device' error below onmultiple gluster v10.4 nodes in the logs. At the moment seeing it in thelogs for about half (5 out of 12) of the nodes. The issue will go away ifwe reboot all the glusterfs nodes but, backups take a little over 2 days tocomplete each weekend and the issue returns after about 1 day of backupsrunning and before the backup cycle is complete. It has been happening thelast 2 weekends we have run backups to these nodes.#example log msg from /var/log/glusterfs/home-volbackups.log[2023-05-01 21:43:15.450502 +0000] W [MSGID: 114031][client-rpc-fops_v2.c:670:client4_0_writev_cbk] 0-volbackups-client-18:remote operation failed. [{errno=28}, {error=No space left on device}]Each glusterfs node has a single brick and mounts a single distributedvolume as a glusterfs client locally and receives backup files to the volumeeach weekend.We distribute the ftp upload load between the servers through a combinationof /etc/hosts entries and AWS weighted dns.We have 91 TB available on the volume though and each of the 12 nodes have4-11 TB free so we are nowhere near out of space on any node?We have already tried the setting change from 'cluster.min-free-disk: 1%' to'cluster.min-free-disk: 1GB' and rebooted all the gluster nodes to refreshthem and it happened again. That was mentioned in this dochttps://access.redhat.com/solutions/276483 as an idea.Does anyone know what we might check next?glusterfs-server-10.4-1.el8s.x86_64glusterfs-fuse-10.4-1.el8s.x86_64Here is the info (hostnames changed) below.[root@nybaknode1 ~]# gluster volume status volbackups detailStatus of volume: volbackups------------------------------------------------------------------------------Brick : Brick nybaknode9.example.net:/lvbackups/brickTCP Port : 60039RDMA Port : 0Online : YPid : 1664File System : xfsDevice : /dev/mapper/vgbackups-lvbackupsMount Options :rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquotaInode Size : 512Disk Space Free : 6.1TBTotal Disk Space : 29.0TBInode Count : 3108974976Free Inodes : 3108881513------------------------------------------------------------------------------Brick : Brick nybaknode11.example.net:/lvbackups/brickTCP Port : 52682RDMA Port : 0Online : YPid : 2076File System : xfsDevice : /dev/mapper/vgbackups-lvbackupsMount Options :rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquotaInode Size : 512Disk Space Free : 10.1TBTotal Disk Space : 43.5TBInode Count : 4672138432Free Inodes : 4672039743------------------------------------------------------------------------------Brick : Brick nybaknode2.example.net:/lvbackups/brickTCP Port : 56722RDMA Port : 0Online : YPid : 1761File System : xfsDevice : /dev/mapper/vgbackups-lvbackupsMount Options :rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquotaInode Size : 512Disk Space Free : 6.6TBTotal Disk Space : 29.0TBInode Count : 3108921344Free Inodes : 3108827241------------------------------------------------------------------------------Brick : Brick nybaknode3.example.net:/lvbackups/brickTCP Port : 53098RDMA Port : 0Online : YPid : 1601File System : xfsDevice : /dev/mapper/vgbackups-lvbackupsMount Options :rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquotaInode Size : 512Disk Space Free : 6.4TBTotal Disk Space : 29.0TBInode Count : 3108921344Free Inodes : 3108827312------------------------------------------------------------------------------Brick : Brick nybaknode4.example.net:/lvbackups/brickTCP Port : 51476RDMA Port : 0Online : YPid : 1633File System : xfsDevice : /dev/mapper/vgbackups-lvbackupsMount Options :rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquotaInode Size : 512Disk Space Free : 6.9TBTotal Disk Space : 29.0TBInode Count : 3108921344Free Inodes : 3108826837------------------------------------------------------------------------------Brick : Brick nybaknode12.example.net:/lvbackups/brickTCP Port : 50224RDMA Port : 0Online : YPid : 1966File System : xfsDevice : /dev/mapper/vgbackups-lvbackupsMount Options :rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquotaInode Size : 512Disk Space Free : 9.9TBTotal Disk Space : 43.5TBInode Count : 4671718976Free Inodes : 4671620385------------------------------------------------------------------------------Brick : Brick nybaknode5.example.net:/lvbackups/brickTCP Port : 55270RDMA Port : 0Online : YPid : 1666File System : xfsDevice : /dev/mapper/vgbackups-lvbackupsMount Options :rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquotaInode Size : 512Disk Space Free : 7.4TBTotal Disk Space : 29.0TBInode Count : 3108921344Free Inodes : 3108827575------------------------------------------------------------------------------Brick : Brick nybaknode6.example.net:/lvbackups/brickTCP Port : 53106RDMA Port : 0Online : YPid : 1688File System : xfsDevice : /dev/mapper/vgbackups-lvbackupsMount Options :rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquotaInode Size : 512Disk Space Free : 7.6TBTotal Disk Space : 29.0TBInode Count : 3108921344Free Inodes : 3108827668------------------------------------------------------------------------------Brick : Brick nybaknode7.example.net:/lvbackups/brickTCP Port : 56734RDMA Port : 0Online : YPid : 1655File System : xfsDevice : /dev/mapper/vgbackups-lvbackupsMount Options :rw,relatime,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=512,noquotaInode Size : 512Disk Space Free : 4.0TBTotal Disk Space : 14.4TBInode Count : 1546333376Free Inodes : 1546245572------------------------------------------------------------------------------Brick : Brick nybaknode8.example.net:/lvbackups/brickTCP Port : 60208RDMA Port : 0Online : YPid : 1754File System : xfsDevice : /dev/mapper/vgbackups-lvbackupsMount Options :rw,relatime,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=128,noquotaInode Size : 512Disk Space Free : 7.0TBTotal Disk Space : 29.0TBInode Count : 3108921344Free Inodes : 3108827378------------------------------------------------------------------------------Brick : Brick nybaknode10.example.net:/lvbackups/brickTCP Port : 53237RDMA Port : 0Online : YPid : 1757File System : xfsDevice : /dev/mapper/vgbackups-lvbackupsMount Options :rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512,noquotaInode Size : 512Disk Space Free : 10.5TBTotal Disk Space : 29.0TBInode Count : 3108921344Free Inodes : 3108828289------------------------------------------------------------------------------Brick : Brick nybaknode1.example.net:/lvbackups/brickTCP Port : 54446RDMA Port : 0Online : YPid : 1685File System : xfsDevice : /dev/mapper/vgbackups-lvbackupsMount Options :rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=128,swidth=128,noquotaInode Size : 512Disk Space Free : 11.2TBTotal Disk Space : 29.0TBInode Count : 3108921344Free Inodes : 3108828701[root@nybaknode1 ~]#[root@nybaknode1 ~]# gluster volume status volbackupsStatus of volume: volbackupsGluster process TCP Port RDMA Port Online Pid------------------------------------------------------------------------------Brick nybaknode9.example.net:/lvbackups/brick 60039 0 Y1664Brick nybaknode11.example.net:/lvbackups/brick 52682 0 Y2076Brick nybaknode2.example.net:/lvbackups/brick 56722 0 Y1761Brick nybaknode3.example.net:/lvbackups/brick 53098 0 Y1601Brick nybaknode4.example.net:/lvbackups/brick 51476 0 Y1633Brick nybaknode12.example.net:/lvbackups/brick 50224 0 Y1966Brick nybaknode5.example.net:/lvbackups/brick 55270 0 Y1666Brick nybaknode6.example.net:/lvbackups/brick 53106 0 Y1688Brick nybaknode7.example.net:/lvbackups/brick 56734 0 Y1655Brick nybaknode8.example.net:/lvbackups/brick 60208 0 Y1754Brick nybaknode10.example.net:/lvbackups/brick 53237 0 Y1757Brick nybaknode1.example.net:/lvbackups/brick 54446 0 Y1685Task Status of Volume volbackups------------------------------------------------------------------------------Task : RebalanceID : df687907-fee4-4a46-9d23-2cfb38cb17cdStatus : completed[root@nybaknode1 ~]#[root@nybaknode1 ~]# gluster volume info volbackupsVolume Name: volbackupsType: DistributeVolume ID: cd40794d-ab74-4706-a0bc-3e95bb8c63a2Status: StartedSnapshot Count: 0Number of Bricks: 12Transport-type: tcpBricks:Brick1: nybaknode9.example.net:/lvbackups/brickBrick2: nybaknode11.example.net:/lvbackups/brickBrick3: nybaknode2.example.net:/lvbackups/brickBrick4: nybaknode3.example.net:/lvbackups/brickBrick5: nybaknode4.example.net:/lvbackups/brickBrick6: nybaknode12.example.net:/lvbackups/brickBrick7: nybaknode5.example.net:/lvbackups/brickBrick8: nybaknode6.example.net:/lvbackups/brickBrick9: nybaknode7.example.net:/lvbackups/brickBrick10: nybaknode8.example.net:/lvbackups/brickBrick11: nybaknode10.example.net:/lvbackups/brickBrick12: nybaknode1.example.net:/lvbackups/brickOptions Reconfigured:cluster.min-free-disk: 1GBnfs.disable: ontransport.address-family: inetperformance.cache-max-file-size: 2MBdiagnostics.brick-log-level: WARNINGdiagnostics.brick-sys-log-level: WARNINGclient.event-threads: 16performance.client-io-threads: onperformance.io-thread-count: 32server.event-threads: 16performance.cache-size: 256MB[root@nybaknode1 ~]#________Community Meeting Calendar:Schedule -Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTCGluster-users mailing list
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
