|
Hello xfs experts,
I am crawling in the dark from few days and I have
no idea how to fix the following problem. On a centos 7 system:
# uname -a
Linux 1a 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux df is reporting 100% full of / and du is reporting
only 1.7G usage from 50GB available (less than 4%). I want to mention that / is
xfs. See below:
# df -a|grep ^/
/dev/mapper/centos-root 52403200 52400396 2804 100% / ^^^^^^^^^^ ^^^^^^^^^^ /dev/sda1 503040 131876 371164 27% /boot /dev/mapper/centos-home 210529792 35204 210494588 1% /home du is estimating just 1.7G usage of /
# du -sch /* --exclude=home --exclude=boot 0 /bin 0 /dev 25M /etc 0 /lib 0 /lib64 744K /luarocks-2.3.0 0 /media 0 /mnt 125M /openresty-1.9.7.4 0 /opt 420K /root 49M /run 0 /sbin 0 /srv 0 /sys 0 /tmp 1.3G /usr 227M /var 1.7G total [root@localhost ~]# df is also reporting 80% of inode
usage:
# df
-i
Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/centos-root 78160 66218 11942 85% / ^^^^^^^^ devtmpfs 8218272 519 8217753 1% /dev tmpfs 8221010 1 8221009 1% /dev/shm tmpfs 8221010 648 8220362 1% /run tmpfs 8221010 13 8220997 1% /sys/fs/cgroup /dev/sda1 509952 330 509622 1% /boot /dev/mapper/centos-home 210632704 99 210632605 1% /home tmpfs 8221010 1 8221009 1% /run/user/0 # / partition is created on top of a LVM having also
50GB size.
# lvdisplay /dev/centos/root
--- Logical volume --- LV Path /dev/centos/root LV Name root VG Name centos LV
Status
available
# open 1 LV Size 50.00 GiB Current LE 12800 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0 I've already checked against rootkit without
finding anything wrong!
I have another system, identical with this one
which is healthy. The only difference I found between those systems is regarding
max number of inodes available on / (which has the same size, 50GB on booth
servers). On the second one (healthy), max number of inodes are ~52 milions and
not only just ~85.000 as are reported on "seek" server.
# df -i|grep ^/
/dev/mapper/centos-root 52424704 66137 52358567 1% / ^^^^^^^^^^^^^ /dev/sda1 509952 330 509622 1% /boot /dev/mapper/centos-home 210632704 26 210632678 1% /home [root@localhost ~]# Suspected also large number of files on /. Counted
total number of files and or booth servers are the same: ~180K. So no difference
here.
Look to find also files larger than 100M and on
booth servers and found just 1 (104M size):
find / -type f -size +100000k -exec ls -lh {}
\;
# /usr/lib/locale/locale-archive # Looking to find files larger than 10M, I found just
~20 on booth servers.
# find / -type f -size +10000k -exec ls -lh {} \;
|wc -l
16 # So for sure, there are NO files exhausting free
space.
On booth servers, number of used inodes are
identical: ~66K. Also xfs_info report is identical for booth. What is different
is number of AVAILABLE inodes: 85K (on seek node) vs 52 milion (on healthy
node)!!! How is possible that!!! Booth servers has the same size (50GB) for
/!
#lsof -nP |grep -i delete|wc -l
0 #find /proc/*/fd -ls | grep -i dele|wc -l 0 so lsof and find does not report anything wrong
(any file deleted and still open)!
reboot does not fix the problem, / remain 100%
full
After reboot, on 25th July:
# df -ah|grep
centos-root
/dev/mapper/centos-root 50G 50G 4.0M 100% / # Also max number of inodes = 67k:
# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/centos-root 66960 66165 795 99% / ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ devtmpfs 8218272 519 8217753 1% /dev tmpfs 8221010 1 8221009 1% /dev/shm tmpfs 8221010 630 8220380 1% /run tmpfs 8221010 13 8220997 1% /sys/fs/cgroup /dev/sda1 509952 330 509622 1% /boot /dev/mapper/centos-home 210632704 28 210632676 1% /home tmpfs 8221010 1 8221009 1% /run/user/0 # Lets try to run intentionally xfs_grow (which
normally should not produce any change)
# xfs_growfs
/dev/mapper/centos-root
meta-data="" isize=256 agcount=16, agsize=819136 blks = sectsz=512 attr=2, projid32bit=1 = crc=0 finobt=0 data = bsize=4096 blocks=13106176, imaxpct=25 = sunit=64 swidth=64 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=0 log =internal bsize=4096 blocks=6400, version=2 = sectsz=512 sunit=64 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 data blocks changed from 13106176 to 13107200 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ # Partition remain the same, 50GB
size:
[root@nl-hvs-ov001a ~]# df -ah|grep centos-root /dev/mapper/centos-root 50G 50G 4.0M 100% / But number of inodes INCREASED with more tha
20%!!!
# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/centos-root 83200 66165 17035 80% / ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ devtmpfs 8218272 519 8217753 1% /dev tmpfs 8221010 1 8221009 1% /dev/shm tmpfs 8221010 630 8220380 1% /run tmpfs 8221010 13 8220997 1% /sys/fs/cgroup /dev/sda1 509952 330 509622 1% /boot /dev/mapper/centos-home 210632704 28 210632676 1% /home tmpfs 8221010 1 8221009 1% /run/user/0 # On 27July without changing anything there, max
number inodes available for / decreased to ~67k (the same size like 2 days ago,
before xfs_grow)!
# df
-i
Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/centos-root 67024 66225 799 99% / ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ devtmpfs 8218272 519 8217753 1% /dev tmpfs 8221010 1 8221009 1% /dev/shm tmpfs 8221010 632 8220378 1% /run tmpfs 8221010 13 8220997 1% /sys/fs/cgroup /dev/mapper/centos-home 210632704 99 210632605 1% /home /dev/sda1 509952 330 509622 1% /boot tmpfs 8221010 1 8221009 1% /run/user/0 # Please note that all that time, number of files
remain unchanged ~180K, the same for inodes used, the number remain constant
~66K. Just max number of inodes available decreased which is an abnormal
behavior.
How can be fixed? Looks like xfs is crrupted
or like a bug.
Thanks in advance for help.
Alex |
_______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
