partition 100% full No space left on device. looks like xfs is corrupted or a bug

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello xfs experts,
 
I am crawling in the dark from few days and I have no idea how to fix the following problem. On a centos 7 system:
 
# uname -a
Linux 1a 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
 
df is reporting 100% full of / and du is reporting only 1.7G usage from 50GB available (less than 4%). I want to mention that / is xfs. See below:
 
# df -a|grep ^/
/dev/mapper/centos-root  52403200 52400396      2804 100% /
                                     ^^^^^^^^^^   ^^^^^^^^^^
/dev/sda1                  503040   131876    371164  27% /boot
/dev/mapper/centos-home 210529792    35204 210494588   1% /home
 
du is estimating just 1.7G usage of /
# du -sch /* --exclude=home --exclude=boot
0       /bin
0       /dev
25M     /etc
0       /lib
0       /lib64
744K    /luarocks-2.3.0
0       /media
0       /mnt
125M    /openresty-1.9.7.4
0       /opt
420K    /root
49M     /run
0       /sbin
0       /srv
0       /sys
0       /tmp
1.3G    /usr
227M    /var
1.7G    total
[root@localhost ~]#
 
df is also reporting 80% of inode usage:
 
# df -i
Filesystem                 Inodes IUsed     IFree IUse% Mounted on
/dev/mapper/centos-root     78160 66218     11942   85% /
                                       ^^^^^^^^
devtmpfs                  8218272   519   8217753    1% /dev
tmpfs                     8221010     1   8221009    1% /dev/shm
tmpfs                     8221010   648   8220362    1% /run
tmpfs                     8221010    13   8220997    1% /sys/fs/cgroup
/dev/sda1                  509952   330    509622    1% /boot
/dev/mapper/centos-home 210632704    99 210632605    1% /home
tmpfs                     8221010     1   8221009    1% /run/user/0
#
 
/ partition is created on top of a LVM having also 50GB size.
 
# lvdisplay /dev/centos/root
  --- Logical volume ---
  LV Path                /dev/centos/root
  LV Name                root
  VG Name                centos
 
  LV Status              available
  # open                 1
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
 
I've already checked against rootkit without finding anything wrong!
 
I have another system, identical with this one which is healthy. The only difference I found between those systems is regarding max number of inodes available on / (which has the same size, 50GB on booth servers). On the second one (healthy), max number of inodes are ~52 milions and not only just ~85.000 as are reported on "seek" server.
 
# df -i|grep ^/
/dev/mapper/centos-root  52424704 66137  52358567    1% /
                                   ^^^^^^^^^^^^^
/dev/sda1                                  509952   330    509622    1% /boot
/dev/mapper/centos-home 210632704    26 210632678    1% /home
[root@localhost ~]#
 
Suspected also large number of files on /. Counted total number of files and or booth servers are the same: ~180K. So no difference here.
 
Look to find also files larger than 100M and on booth servers and found just 1 (104M size):
 
find / -type f -size +100000k -exec ls -lh {} \;
#
/usr/lib/locale/locale-archive
#
 
Looking to find files larger than 10M, I found just ~20 on booth servers.
 
# find / -type f -size +10000k -exec ls -lh {} \; |wc -l
16
#
 
So for sure, there are NO files exhausting free space.
 
On booth servers, number of used inodes are identical: ~66K. Also xfs_info report is identical for booth. What is different is number of AVAILABLE inodes: 85K (on seek node) vs 52 milion (on healthy node)!!! How is possible that!!! Booth servers has the same size (50GB) for /!
 
#lsof -nP |grep -i delete|wc -l
0
#find /proc/*/fd -ls | grep -i dele|wc -l
0
 
so lsof and find does not report anything wrong (any file deleted and still open)!
 
reboot does not fix the problem, / remain 100% full
 
After reboot, on 25th July:
 
# df -ah|grep centos-root
/dev/mapper/centos-root   50G   50G  4.0M 100% /
#
 
Also max number of inodes = 67k:
# df -i
Filesystem                 Inodes IUsed     IFree IUse% Mounted on
/dev/mapper/centos-root     66960 66165       795   99% /
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
devtmpfs                  8218272   519   8217753    1% /dev
tmpfs                     8221010     1   8221009    1% /dev/shm
tmpfs                     8221010   630   8220380    1% /run
tmpfs                     8221010    13   8220997    1% /sys/fs/cgroup
/dev/sda1                  509952   330    509622    1% /boot
/dev/mapper/centos-home 210632704    28 210632676    1% /home
tmpfs                     8221010     1   8221009    1% /run/user/0
#
 
Lets try to run intentionally xfs_grow (which normally should not produce any change)
 
# xfs_growfs /dev/mapper/centos-root
meta-data="" isize=256    agcount=16, agsize=819136 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=13106176, imaxpct=25
         =                       sunit=64     swidth=64 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=6400, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 13106176 to 13107200
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#
 
Partition remain the same, 50GB size:
[root@nl-hvs-ov001a ~]# df -ah|grep centos-root
/dev/mapper/centos-root   50G   50G  4.0M 100% /
 
But number of inodes INCREASED with more tha 20%!!!
# df -i
Filesystem                 Inodes IUsed     IFree IUse% Mounted on
/dev/mapper/centos-root     83200 66165     17035   80% /
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
devtmpfs                  8218272   519   8217753    1% /dev
tmpfs                     8221010     1   8221009    1% /dev/shm
tmpfs                     8221010   630   8220380    1% /run
tmpfs                     8221010    13   8220997    1% /sys/fs/cgroup
/dev/sda1                  509952   330    509622    1% /boot
/dev/mapper/centos-home 210632704    28 210632676    1% /home
tmpfs                     8221010     1   8221009    1% /run/user/0
#
 
On 27July without changing anything there, max number inodes available for / decreased to ~67k (the same size like 2 days ago, before xfs_grow)!
 
# df -i
Filesystem                 Inodes IUsed     IFree IUse% Mounted on
/dev/mapper/centos-root     67024 66225       799   99% /
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
devtmpfs                  8218272   519   8217753    1% /dev
tmpfs                     8221010     1   8221009    1% /dev/shm
tmpfs                     8221010   632   8220378    1% /run
tmpfs                     8221010    13   8220997    1% /sys/fs/cgroup
/dev/mapper/centos-home 210632704    99 210632605    1% /home
/dev/sda1                  509952   330    509622    1% /boot
tmpfs                     8221010     1   8221009    1% /run/user/0
#
 
Please note that all that time, number of files remain unchanged ~180K, the same for inodes used, the number remain constant ~66K. Just max number of inodes available decreased which is an abnormal behavior.
 
How can be fixed? Looks like xfs is crrupted or like a bug.
 
Thanks in advance for help.
Alex
 
_______________________________________________
xfs mailing list
xfs@xxxxxxxxxxx
http://oss.sgi.com/mailman/listinfo/xfs

[Index of Archives]     [Linux XFS Devel]     [Linux Filesystem Development]     [Filesystem Testing]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux