On Fri, Jul 29, 2016 at 12:01:42PM +0300, Lista Unx wrote:
> Hello xfs experts,
>
> I am crawling in the dark from few days and I have no idea how to fix the following problem. On a centos 7 system:
>
> # uname -a
> Linux 1a 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> df is reporting 100% full of / and du is reporting only 1.7G usage from 50GB available (less than 4%). I want to mention that / is xfs. See below:
>
First and foremost, have you run 'xfs_repair -n' to see if the fs is
healthy? If so, the next thing I would probably try is mount from a
single user mode of some sort (or boot a livecd) and recheck from there
to rule out any OS runtime weirdness going on (open but unlinked files,
files hidden under mount points, etc.).
Brian
> # df -a|grep ^/
> /dev/mapper/centos-root 52403200 52400396 2804 100% /
> ^^^^^^^^^^ ^^^^^^^^^^
> /dev/sda1 503040 131876 371164 27% /boot
> /dev/mapper/centos-home 210529792 35204 210494588 1% /home
>
> du is estimating just 1.7G usage of /
> # du -sch /* --exclude=home --exclude=boot
> 0 /bin
> 0 /dev
> 25M /etc
> 0 /lib
> 0 /lib64
> 744K /luarocks-2.3.0
> 0 /media
> 0 /mnt
> 125M /openresty-1.9.7.4
> 0 /opt
> 420K /root
> 49M /run
> 0 /sbin
> 0 /srv
> 0 /sys
> 0 /tmp
> 1.3G /usr
> 227M /var
> 1.7G total
> [root@localhost ~]#
>
> df is also reporting 80% of inode usage:
>
> # df -i
> Filesystem Inodes IUsed IFree IUse% Mounted on
> /dev/mapper/centos-root 78160 66218 11942 85% /
> ^^^^^^^^
> devtmpfs 8218272 519 8217753 1% /dev
> tmpfs 8221010 1 8221009 1% /dev/shm
> tmpfs 8221010 648 8220362 1% /run
> tmpfs 8221010 13 8220997 1% /sys/fs/cgroup
> /dev/sda1 509952 330 509622 1% /boot
> /dev/mapper/centos-home 210632704 99 210632605 1% /home
> tmpfs 8221010 1 8221009 1% /run/user/0
> #
>
> / partition is created on top of a LVM having also 50GB size.
>
> # lvdisplay /dev/centos/root
> --- Logical volume ---
> LV Path /dev/centos/root
> LV Name root
> VG Name centos
>
> LV Status available
> # open 1
> LV Size 50.00 GiB
> Current LE 12800
> Segments 1
> Allocation inherit
> Read ahead sectors auto
> - currently set to 256
> Block device 253:0
>
> I've already checked against rootkit without finding anything wrong!
>
> I have another system, identical with this one which is healthy. The only difference I found between those systems is regarding max number of inodes available on / (which has the same size, 50GB on booth servers). On the second one (healthy), max number of inodes are ~52 milions and not only just ~85.000 as are reported on "seek" server.
>
> # df -i|grep ^/
> /dev/mapper/centos-root 52424704 66137 52358567 1% /
> ^^^^^^^^^^^^^
> /dev/sda1 509952 330 509622 1% /boot
> /dev/mapper/centos-home 210632704 26 210632678 1% /home
> [root@localhost ~]#
>
> Suspected also large number of files on /. Counted total number of files and or booth servers are the same: ~180K. So no difference here.
>
> Look to find also files larger than 100M and on booth servers and found just 1 (104M size):
>
> find / -type f -size +100000k -exec ls -lh {} \;
> #
> /usr/lib/locale/locale-archive
> #
>
> Looking to find files larger than 10M, I found just ~20 on booth servers.
>
> # find / -type f -size +10000k -exec ls -lh {} \; |wc -l
> 16
> #
>
> So for sure, there are NO files exhausting free space.
>
> On booth servers, number of used inodes are identical: ~66K. Also xfs_info report is identical for booth. What is different is number of AVAILABLE inodes: 85K (on seek node) vs 52 milion (on healthy node)!!! How is possible that!!! Booth servers has the same size (50GB) for /!
>
> #lsof -nP |grep -i delete|wc -l
> 0
> #find /proc/*/fd -ls | grep -i dele|wc -l
> 0
>
> so lsof and find does not report anything wrong (any file deleted and still open)!
>
> reboot does not fix the problem, / remain 100% full
>
> After reboot, on 25th July:
>
> # df -ah|grep centos-root
> /dev/mapper/centos-root 50G 50G 4.0M 100% /
> #
>
> Also max number of inodes = 67k:
> # df -i
> Filesystem Inodes IUsed IFree IUse% Mounted on
> /dev/mapper/centos-root 66960 66165 795 99% /
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> devtmpfs 8218272 519 8217753 1% /dev
> tmpfs 8221010 1 8221009 1% /dev/shm
> tmpfs 8221010 630 8220380 1% /run
> tmpfs 8221010 13 8220997 1% /sys/fs/cgroup
> /dev/sda1 509952 330 509622 1% /boot
> /dev/mapper/centos-home 210632704 28 210632676 1% /home
> tmpfs 8221010 1 8221009 1% /run/user/0
> #
>
> Lets try to run intentionally xfs_grow (which normally should not produce any change)
>
> # xfs_growfs /dev/mapper/centos-root
> meta-data=/dev/mapper/centos-root isize=256 agcount=16, agsize=819136 blks
> = sectsz=512 attr=2, projid32bit=1
> = crc=0 finobt=0
> data = bsize=4096 blocks=13106176, imaxpct=25
> = sunit=64 swidth=64 blks
> naming =version 2 bsize=4096 ascii-ci=0 ftype=0
> log =internal bsize=4096 blocks=6400, version=2
> = sectsz=512 sunit=64 blks, lazy-count=1
> realtime =none extsz=4096 blocks=0, rtextents=0
> data blocks changed from 13106176 to 13107200
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> #
>
> Partition remain the same, 50GB size:
> [root@nl-hvs-ov001a ~]# df -ah|grep centos-root
> /dev/mapper/centos-root 50G 50G 4.0M 100% /
>
> But number of inodes INCREASED with more tha 20%!!!
> # df -i
> Filesystem Inodes IUsed IFree IUse% Mounted on
> /dev/mapper/centos-root 83200 66165 17035 80% /
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> devtmpfs 8218272 519 8217753 1% /dev
> tmpfs 8221010 1 8221009 1% /dev/shm
> tmpfs 8221010 630 8220380 1% /run
> tmpfs 8221010 13 8220997 1% /sys/fs/cgroup
> /dev/sda1 509952 330 509622 1% /boot
> /dev/mapper/centos-home 210632704 28 210632676 1% /home
> tmpfs 8221010 1 8221009 1% /run/user/0
> #
>
> On 27July without changing anything there, max number inodes available for / decreased to ~67k (the same size like 2 days ago, before xfs_grow)!
>
> # df -i
> Filesystem Inodes IUsed IFree IUse% Mounted on
> /dev/mapper/centos-root 67024 66225 799 99% /
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> devtmpfs 8218272 519 8217753 1% /dev
> tmpfs 8221010 1 8221009 1% /dev/shm
> tmpfs 8221010 632 8220378 1% /run
> tmpfs 8221010 13 8220997 1% /sys/fs/cgroup
> /dev/mapper/centos-home 210632704 99 210632605 1% /home
> /dev/sda1 509952 330 509622 1% /boot
> tmpfs 8221010 1 8221009 1% /run/user/0
> #
>
> Please note that all that time, number of files remain unchanged ~180K, the same for inodes used, the number remain constant ~66K. Just max number of inodes available decreased which is an abnormal behavior.
>
> How can be fixed? Looks like xfs is crrupted or like a bug.
>
> Thanks in advance for help.
> Alex
> _______________________________________________
> xfs mailing list
> xfs@xxxxxxxxxxx
> http://oss.sgi.com/mailman/listinfo/xfs
_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs
