On Wed 26-06-19 11:17:54, Thomas Walker wrote: > Sorry to revive a rather old thread, but Elana mentioned that there might > have been a related fix recently? Possibly something to do with > truncate? A quick scan of the last month or so turned up > https://www.spinics.net/lists/linux-ext4/msg65772.html but none of these > seemed obviously applicable to me. We do still experience this phantom > space usage quite frequently (although the remount workaround below has > lowered the priority). I don't recall any fix for this. But seeing that remount "fixes" the issue for you can you try whether one of the following has a similar effect? 1) Try "sync" 2) Try "fsfreeze -f / && fsfreeze -u /" 3) Try "echo 3 >/proc/sys/vm/drop_caches" Also what is the contents of /sys/fs/ext4/<problematic-device>/delayed_allocation_blocks when the issue happens? Honza > > On Wed, Jan 23, 2019 at 02:59:22PM -0500, Thomas Walker wrote: > > Unfortunately this still continues to be a persistent problem for us. On another example system: > > > > # uname -a > > Linux <hostname> 4.14.67-ts1 #1 SMP Wed Aug 29 13:28:25 UTC 2018 x86_64 GNU/Linux > > > > # df -h / > > Filesystem Size Used Avail Use% Mounted on > > /dev/disk/by-uuid/<uuid> 50G 46G 1.1G 98% / > > > > # df -hi / > > Filesystem Inodes IUsed IFree IUse% Mounted on > > /dev/disk/by-uuid/<uuid> 3.2M 306K 2.9M 10% / > > > > # du -hsx / > > 14G / > > > > And confirmed not to be due to sparse files or deleted but still open files. > > > > The most interesting thing that I've been able to find so far is this: > > > > # mount -o remount,ro / > > mount: / is busy > > # df -h / > > Filesystem Size Used Avail Use% Mounted on > > /dev/disk/by-uuid/<uuid> 50G 14G 33G 30% / > > > > Something about attempting (and failing) to remount read-only frees up all of the phantom space usage. > > Curious whether that sparks ideas in anyone's mind? > > > > I've tried all manner of other things without success. Unmounting all of the overlays. Killing off virtually all of usersapce (dropping to single user). Dropping page/inode/dentry caches.Nothing else (short of a reboot) seems to give us the space back. -- Jan Kara <jack@xxxxxxxx> SUSE Labs, CR
