All,
Recently we ran into a problem where our filesystem (300GB in size) reported 'no space left on device' (ENOSPC) but when we looked at disk space usage and inode usage it was around 52% for disk space and 11% for inode. (sorry do not have a save of the output of df command).
Second we looked at the xfs_info and xfs_db -c freesp for each AG. Sadly we do not have a copy anymore of the xfs_info from that time but I do have that of the freesp (at end of this mail).
As we did not fully understand the output of the 'xfs_db -c freesp' we simply added another 100GB to the filesystem and our problem was solved for now. The AG count went from 4 to 6.
For now we are looking into how can we detect (monitor) this filesystem and see the same problem creeping up to us before it happens.
After searching online, reading a lot of mail threads and info pages it looks like we hit the problem of not having any continuous blocks of at least 16k to hold a new inode (http://oss.sgi.com/pipermail/xfs/2014-September/038301.html).
Currently the xfs_info (after adding 100GB) looks like this:
meta-data="" isize=256 agcount=6, agsize=19660800 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=104857600, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=38400, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
The output of 'xfs_db -c freesp' shows a historgram. We first thought the "from" column shows the minimal number of continous blocks. And as each block is 4K (4096) the row with 'from 4 to 7' contains at least 4 continuous blocks with a minimal size of 4x4k=16k. So our filesystem has enough blocks and extents which are 16k for new inodes, atleast what we think.
- So why did we still get the ENOSPC error?
- Where can I find a better explanation of the 'xfs_db -c freesp' output?
- What is the best way to monitor/detect this problem before it happens.
Some extra info about the system and files on the system:
- Ubuntu 12.04.5 LTS
- Kernel: Linux ealxs00170 3.2.0-97-generic #137-Ubuntu SMP Thu Dec 17 18:11:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
- 32 million files of which about 75% is smaller than 1k. Files are separated over different folders to keep the number of files per folder low.
- mount options: defaults,noatime,inode64,nobarrier Recently we ran into a problem where our filesystem (300GB in size) reported 'no space left on device' (ENOSPC) but when we looked at disk space usage and inode usage it was around 52% for disk space and 11% for inode. (sorry do not have a save of the output of df command).
Second we looked at the xfs_info and xfs_db -c freesp for each AG. Sadly we do not have a copy anymore of the xfs_info from that time but I do have that of the freesp (at end of this mail).
As we did not fully understand the output of the 'xfs_db -c freesp' we simply added another 100GB to the filesystem and our problem was solved for now. The AG count went from 4 to 6.
For now we are looking into how can we detect (monitor) this filesystem and see the same problem creeping up to us before it happens.
After searching online, reading a lot of mail threads and info pages it looks like we hit the problem of not having any continuous blocks of at least 16k to hold a new inode (http://oss.sgi.com/pipermail/xfs/2014-September/038301.html).
Currently the xfs_info (after adding 100GB) looks like this:
meta-data="" isize=256 agcount=6, agsize=19660800 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=104857600, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=38400, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
The output of 'xfs_db -c freesp' shows a historgram. We first thought the "from" column shows the minimal number of continous blocks. And as each block is 4K (4096) the row with 'from 4 to 7' contains at least 4 continuous blocks with a minimal size of 4x4k=16k. So our filesystem has enough blocks and extents which are 16k for new inodes, atleast what we think.
- So why did we still get the ENOSPC error?
- Where can I find a better explanation of the 'xfs_db -c freesp' output?
- What is the best way to monitor/detect this problem before it happens.
Some extra info about the system and files on the system:
- Ubuntu 12.04.5 LTS
- Kernel: Linux ealxs00170 3.2.0-97-generic #137-Ubuntu SMP Thu Dec 17 18:11:47 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
- 32 million files of which about 75% is smaller than 1k. Files are separated over different folders to keep the number of files per folder low.
Regards,
Michel Verbraak.
Store 03 (root@server):~# xfs_db -c freesp /dev/sdb
from to extents blocks pct
1 1 3282633 3282633 9.03
2 3 3416223 8372325 23.03
4 7 6175009 24700036 67.94
Store 03 (root@server):~# xfs_db -c "freesp -a 0" /dev/sdb
from to extents blocks pct
1 1 813491 813491 8.95
2 3 806828 1953270 21.50
4 7 1579585 6318340 69.55
Store 03 (root@server):~# xfs_db -c "freesp -a 1" /dev/sdb
from to extents blocks pct
1 1 835174 835174 9.20
2 3 953626 2381033 26.22
4 7 1465787 5863148 64.58
Store 03 (root@server):~# xfs_db -c "freesp -a 2" /dev/sdb
from to extents blocks pct
1 1 819098 819098 9.05
2 3 846158 2074235 22.93
4 7 1538593 6154372 68.02
Store 03 (root@server):~# xfs_db -c "freesp -a 3" /dev/sdb
from to extents blocks pct
1 1 814870 814870 8.91
2 3 809611 1963787 21.48
4 7 1591044 6364176 69.61
_______________________________________________ xfs mailing list xfs@xxxxxxxxxxx http://oss.sgi.com/mailman/listinfo/xfs
