On Tue, Jul 10, 2018 at 04:39:26PM -0500, Eric Sandeen wrote: > On 7/10/18 8:43 AM, Filippo Giunchedi wrote: > > Hello, > > a little background: at Wikimedia Foundation we are running a 30-hosts > > Openstack Swift cluster to host user media uploads, each host has 12 > > spinning disks formatted individually with xfs. > > > > Some of the recently-formatted filesystems have started reporting > > negative usage upon hitting around 70% usage, though some filesystems > > on the same host kept reporting as expected: > > > > /dev/sdn1 3.7T -14T 17T - /srv/swift-storage/sdn1 > > /dev/sdh1 3.7T -13T 17T - /srv/swift-storage/sdh1 > > /dev/sdc1 3.7T 3.0T 670G 83% /srv/swift-storage/sdc1 > > /dev/sdk1 3.7T 3.1T 643G 83% /srv/swift-storage/sdk1 > > > > We have experienced this bug only on the last four machines to be put > > in service and formatted with xfsprogs 4.9.0+nmu1 from Debian Stretch. > > The remaining hosts were formatted in the past with xfsprogs 3.2.1 or > > older. > > We have also a standby cluster in another datacenter with similar > > configuration and hosts that received write traffic only but not read > > traffic; the standby cluster hasn't experienced the bug and all > > filesystems report the correct usage. > > As far as I can tell the difference in xfsprogs version used for > > formatting means defaults have changed, (e.g. crc is enabled on the > > affected filesystems). Have you seen this issue before and do you know > > how to fix it? > > > > I would love to help debugging this issue, we've been detailing the > > work done so far at https://phabricator.wikimedia.org/T199198 > > What kernel are the problematic nodes running? > > From your repair output: > > root@ms-be1040:~# xfs_repair -n /dev/sde1 > Phase 1 - find and verify superblock... > Phase 2 - using internal log > - zero log... > - scan filesystem freespace and inode maps... > sb_fdblocks 4461713825, counted 166746529 > - found root inode chunk > > that sb_fdblocks really is ~17T which indicates the problem > really is on disk. > > 4461713825 > 100001001111100000101100110100001 > 166746529 > 1001111100000101100110100001 > > you have a bit flipped in the problematic value... but you're running > with CRCs so it seems unlikely to have been some sort of bit-rot (that, > and the fact that you're hitting the same problem on multiple nodes). > > Soooo not sure what to say right now other than "your bad value has an > extra bit set for some reason." Looks like the superblock verifier doesn't bounds check free block or free/used inode counts. Perhaps we should be checking this in the verifier so in-memory corruption like this never makes it to disk? Cheers, Dave. -- Dave Chinner david@xxxxxxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
