On 17/10/2018 11.47, Christoph Hellwig wrote:
On Wed, Oct 17, 2018 at 10:52:48AM +0300, Avi Kivity wrote:
I have a user running a 1.7TB filesystem with ~10% usage (as shown by df),
getting sporadic ENOSPC errors. The disk is mounted with inode64 and has a
relatively small number of large files. The disk is a single-member RAID0
array, with 1MB chunk size. There are 32 AGs. Running Linux 4.9.17.
4.9.17 is rather old and you'll have a hard time finding someone
familiar with it..
Yes. I expect my user will agree to upgrade, but I'd like to recommend
this only if we know there was a real issue and it was resolved, not on
general principles.
Is this a known issue? Would upgrading the kernel help?
Two things that come to mind:
- are you sure there is no open fd to the unlinked files? That would
keep the space allocated until the last link is dropped.
"df" would report that space as occupied, no?
I believe a colleague verified there were no deleted files but I'm not
100% sure.
- even once we drop the inode the space only becomes available once
the transaction has committed. We do force the log if we found
a busy extent, but there might be some issues. Try seeing if you
hit the xfs_extent_busy_force trace point with your workload.
I'll ask permission to check this and report.
- if you have online discard (-o discard) enabled there might be
more issues like the above, especially on old kernels.
Online discard is not enabled:
/dev/md0 on /var/lib/scylla type xfs
(rw,noatime,attr2,inode64,sunit=2048,swidth=2048,noquota)
btw, we've seen fstrim on an old disk (that was likely never trimmed)
improving its performance by a factor of ~100, so my interest in -o
discard is re-awakening. Is it good enough now to to run on aio
workloads (assuming nvme) or is more work needed? My prime concern is to
avoid io_submit sleeping.
