Hi All,
We had a host starting to fail processing on an ext4 filesystem directly
after extend from 60.5TB to 64TB (lvresize -L64T /dev/lvm/home,
resize2fs /dev/lvm/home).
We rebooted, and now the filesystem will mount but the problem
persists. We've now umounted the filesystem, and fsck complains as follows:
crowsnest ~ # fsck.ext4 -f /dev/lvm/home
e2fsck 1.43.6 (29-Aug-2017)
Superblock has an invalid journal (inode 8).
Clear<y>? yes
*** journal has been deleted ***
Corruption found in superblock. (inodes_count = 0).
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem. If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
or
e2fsck -b 32768 <device>
Corruption found in superblock. (first_ino = 11).
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem. If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
or
e2fsck -b 32768 <device>
Inode count in superblock is 0, should be 4294967295.
Fix<y>? yes
/dev/lvm/home: ***** FILE SYSTEM WAS MODIFIED *****
Note that in spite of supposedly fixing the errors if we re-run fsck the
exact errors repeat. Even if we use -b 32768 (4K blocks) it still repeats.
crowsnest ~ # dumpe2fs -f /dev/lvm/home
dumpe2fs 1.43.6 (29-Aug-2017)
Filesystem volume name: <none>
Last mounted on: /home
Filesystem UUID: 9f0e94bc-25b7-44b7-afdd-bfaa90dbf25c
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr dir_index filetype
meta_bg extent 64bit flex_bg sparse_super large_file huge_file uninit_bg
dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean with errors
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 0
Block count: 17179869184
Reserved block count: 0
Free blocks: 997705247
Free inodes: 4176704883
First block: 0
Block size: 4096
Fragment size: 4096
Group descriptor size: 64
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 512
RAID stride: 128
RAID stripe width: 512
First meta block group: 1472
Flex block group size: 16
Filesystem created: Tue Feb 10 12:51:13 2015
Last mount time: Mon May 21 13:44:11 2018
Last write time: Mon May 21 13:44:27 2018
Mount count: 34
Maximum mount count: -1
Last checked: Sat Aug 19 19:16:38 2017
Check interval: 0 (<none>)
Lifetime writes: 86 TB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: e4a234f2-63e8-4bdd-9591-ba54453485cc
Journal backup: inode blocks
FS Error count: 30230
First error time: Mon May 21 12:47:00 2018
First error function: ext4_search_dir
First error line #: 1296
First error inode #: 304881794
First error block #: 1219511409
Last error time: Mon May 21 13:44:19 2018
Last error function: htree_dirblock_to_tree
Last error line #: 1006
Last error inode #: 2
Last error block #: 9697
dumpe2fs: Illegal inode number while reading journal inode
The filesystem does mount but dmesg gives:
[ 3112.080745] EXT4-fs (dm-5): warning: mounting fs with errors, running
e2fsck is recommended
[ 3112.646029] EXT4-fs (dm-5): mounted filesystem with ordered data
mode. Opts: delalloc,inode_readahead_blks=4096
[ 3120.230898] EXT4-fs error (device dm-5): htree_dirblock_to_tree:1006:
inode #2: block 9697: comm ls: bad entry in directory: inode out of
bounds - offset=0(0), inode=2, rec_len=12, name_len=1
The last line is as soon as we run ls on the mount point (which ends up
giving an empty, not even . or .., listing). ls doesn't give nor does
strace show any errors coming back on any system calls:
stat("/home/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/home/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
getdents(3, /* 0 entries */, 32768) = 0
close(3) = 0
close(1) = 0
close(2) = 0
Interestingly stat does work:
crowsnest ~ # stat /home
File: /home
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fd05h/64773d Inode: 2 Links: 5
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2015-03-28 17:36:47.489839898 +0200
Modify: 2016-12-01 16:52:49.162294603 +0200
Change: 2016-12-01 16:52:49.162294603 +0200
Birth: -
I'm unsure if the filesystem resize completed properly before reboot
.... and also keeping in mind this is LVM on top of software raid (2 x
RAID6 arays consisting of 10 and 12 x 4 TB disks respectively), on top
of an mpt3sas controller. Kernel 4.16 with the patch at
https://www.mail-archive.com/linux-scsi@xxxxxxxxxxxxxxx/msg72812.html
applied (which prevents the system from locking up every few days). As
far as I can determine this patch series has not been mainlined anywhere
in 4.16.*.
Kind Regards,
Jaco
