Hi Ted, We've run into a mysterious "phantom" full filesystem issue on our Kubernetes fleet. We initially encountered this issue on kernel 4.1.35, but are still experiencing the problem after upgrading to 4.14.67. Essentially, `df` reports our root filesystems as full and they behave as though they are full, but the "used" space cannot be accounted for. Rebooting the system, remounting the root filesystem read-only and then remounting as read-write, or booting into single-user mode all free up the "used" space. The disk slowly fills up over time, suggesting that there might be some kind of leak; we previously saw this affecting hosts with ~200 days of uptime on the 4.1 kernel, but are now seeing it affect a 4.14 host with only ~70 days of uptime. Here is some data from an example host, running the 4.14.67 kernel. The root disk is ext4. $ uname -a Linux <hostname> 4.14.67-ts1 #1 SMP Wed Aug 29 13:28:25 UTC 2018 x86_64 GNU/Linux $ grep ' / ' /proc/mounts /dev/disk/by-uuid/<some-uuid> / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0 `df` reports 0 bytes free: $ df -h / Filesystem Size Used Avail Use% Mounted on /dev/disk/by-uuid/<some-uuid> 50G 48G 0 100% / Deleted, open files account for almost no disk capacity: $ sudo lsof -a +L1 / COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME java 5313 user 3r REG 8,3 6806312 0 1315847 /var/lib/sss/mc/passwd (deleted) java 5313 user 11u REG 8,3 55185 0 2494654 /tmp/classpath.1668Gp (deleted) system_ar 5333 user 3r REG 8,3 6806312 0 1315847 /var/lib/sss/mc/passwd (deleted) java 5421 user 3r REG 8,3 6806312 0 1315847 /var/lib/sss/mc/passwd (deleted) java 5421 user 11u REG 8,3 149313 0 2494486 /tmp/java.fzTwWp (deleted) java 5421 tsdist 12u REG 8,3 55185 0 2500513 /tmp/classpath.7AmxHO (deleted) `du` can only account for 16GB of file usage: $ sudo du -hxs / 16G / But what is most puzzling is the numbers reported by e2freefrag, which don't add up: $ sudo e2freefrag /dev/disk/by-uuid/<some-uuid> Device: /dev/disk/by-uuid/<some-uuid> Blocksize: 4096 bytes Total blocks: 13107200 Free blocks: 7778076 (59.3%) Min. free extent: 4 KB Max. free extent: 8876 KB Avg. free extent: 224 KB Num. free extent: 6098 HISTOGRAM OF FREE EXTENT SIZES: Extent Size Range : Free extents Free Blocks Percent 4K... 8K- : 1205 1205 0.02% 8K... 16K- : 980 2265 0.03% 16K... 32K- : 653 3419 0.04% 32K... 64K- : 1337 15374 0.20% 64K... 128K- : 631 14151 0.18% 128K... 256K- : 224 10205 0.13% 256K... 512K- : 261 23818 0.31% 512K... 1024K- : 303 56801 0.73% 1M... 2M- : 387 135907 1.75% 2M... 4M- : 103 64740 0.83% 4M... 8M- : 12 15005 0.19% 8M... 16M- : 2 4267 0.05% This looks like a bug to me; the histogram in the manpage example has percentages that add up to 100% but this doesn't even add up to 5%. After a reboot, `df` reflects real utilization: $ df -h / Filesystem Size Used Avail Use% Mounted on /dev/disk/by-uuid/<some-uuid> 50G 16G 31G 34% / We are using overlay2fs for Docker, as well as rbd mounts; I'm not sure how they might interact. Thanks for your help, -- Elana Hashman ehashman@xxxxxxxxxxxx
