unmounting the fusemount, and running xfs_repair seemed to have solved
the problem.
one remounted and given a HUP for glusterd and glusterfs there are now
more log spews.
the xfs_repair didn't show any errors, so I suspect the problem was with
the Fuse mount and it just needed to be refreshed, it had been up for
hundreds of days.
-wk
On 7/21/23 12:02 PM, W Kern wrote:
we have an older 2+1 arbiter gluster cluster running 6.10 on Ubuntu18LTS
It has run beautifully for years. Only occaisionally needing attention
as drives have died, etc
Each peer has two volumes. G1 and G2 with a shared 'gluster' network.
Since July 1st one of the peers for one volume is spewing the logfile
/var-lib-G1.log with the following errors.
The volume (G2) is not showing this nor are there issue with other
peer and the arbiter for the G1 volume.
So its one machine with one volume that has the problem. There have
been NO issues with the volumes themselves.
It simply a matter of the the logfiles generating GBs of entries every
hour (which is how we noticed it when we started running out of log
space).
According to google there are mentions of this error, but that it was
fixed in the 6.x series. I can find no other mentions.
I have tried restarting glusterd with no change. there doesn't seem to
be any hardware issues.
I am wondering if perhaps this is an XFS file corruption issue and if
I were to unmount the Gluster run xfs_repair and bring it back, that
would solve the issue.
Any other suggestions?
[2023-07-21 18:51:38.260507] W [inode.c:1638:inode_table_prune]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/6.10/xlator/features/shard.so(+0x21b47)
[0x7fb261c13b47]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36)
[0x7fb26947f416]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a)
[0x7fb26947f37a] ) 0-GLB1image-shard: Empty inode lru list found but
with (-2) lru_size
[2023-07-21 18:51:38.261231] W [inode.c:1638:inode_table_prune]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/6.10/xlator/mount/fuse.so(+0xba51)
[0x7fb266cdca51]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36)
[0x7fb26947f416]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a)
[0x7fb26947f37a] ) 0-fuse: Empty inode lru list found but with (-2)
lru_size
[2023-07-21 18:51:38.261377] W [inode.c:1638:inode_table_prune]
(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(loc_wipe+0x12)
[0x7fb26946bd72]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36)
[0x7fb26947f416]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a)
[0x7fb26947f37a] ) 0-GLB1image-shard: Empty inode lru list found but
with (-2) lru_size
[2023-07-21 18:51:38.261806] W [inode.c:1638:inode_table_prune]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/6.10/xlator/cluster/replicate.so(+0x5ca57)
[0x7fb26213ba57]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36)
[0x7fb26947f416]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a)
[0x7fb26947f37a] ) 0-GLB1image-replicate-0: Empty inode lru list found
but with (-2) lru_size
[2023-07-21 18:51:38.261933] W [inode.c:1638:inode_table_prune]
(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(fd_unref+0x1ef)
[0x7fb269495eaf]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36)
[0x7fb26947f416]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a)
[0x7fb26947f37a] ) 0-GLB1image-client-1: Empty inode lru list found
but with (-2) lru_size
[2023-07-21 18:51:38.262684] W [inode.c:1638:inode_table_prune]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/6.10/xlator/cluster/replicate.so(+0x5ca57)
[0x7fb26213ba57]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36)
[0x7fb26947f416]
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a)
[0x7fb26947f37a] ) 0-GLB1image-replicate-0: Empty inode lru list found
but with (-2) lru_size
-wk
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
