You could kill e2fsck and disable the resize_inode feature? There is a different resize mechanism available now (meta_bg) that doesn't need it. Cheers, Andreas > On Mar 16, 2019, at 11:11, Burke Harper <go88team@xxxxxxxxx> wrote: > > I updated to 1.44.5. > > It's been going along now for about 21 hours. There is some > noticeable io this time around. Looks like it's still trying to > recreate the resize inode. > >> On Fri, Mar 15, 2019 at 3:19 PM Andreas Dilger <adilger@xxxxxxxxx> wrote: >> >> Kill your e2fsck and upgrade to the latest version 1.44.5, as it has a lot of fixes over 1.42.13. >> >> If you have the ability, make a "dd" copy of the filesystem first, or a snapshot, and run e2fsck on that first. >> >> Cheers, Andreas >> >>> On Mar 15, 2019, at 00:38, Burke Harper <go88team@xxxxxxxxx> wrote: >>> >>> Over the past weekend, I added 2 more drives to my /dev/md0 array: >>> >>> sudo mdadm --detail /dev/md0 >>> /dev/md0: >>> Version : 1.2 >>> Creation Time : Sat Dec 16 18:32:08 2017 >>> Raid Level : raid6 >>> Array Size : 54697266176 (52163.38 GiB 56010.00 GB) >>> Used Dev Size : 7813895168 (7451.91 GiB 8001.43 GB) >>> Raid Devices : 9 >>> Total Devices : 9 >>> Persistence : Superblock is persistent >>> >>> Intent Bitmap : Internal >>> >>> Update Time : Mon Mar 11 05:13:12 2019 >>> State : clean >>> Active Devices : 9 >>> Working Devices : 9 >>> Failed Devices : 0 >>> Spare Devices : 0 >>> >>> Layout : left-symmetric >>> Chunk Size : 512K >>> >>> Name : powerhouse:0 (local to host powerhouse) >>> UUID : 19b5c7a5:59e4bd00:b4b1c18c:089df9bd >>> Events : 45981 >>> >>> Number Major Minor RaidDevice State >>> 0 8 0 0 active sync /dev/sda >>> 1 8 16 1 active sync /dev/sdb >>> 2 8 32 2 active sync /dev/sdc >>> 3 8 48 3 active sync /dev/sdd >>> 5 8 144 4 active sync /dev/sdj >>> 4 8 128 5 active sync /dev/sdi >>> 6 8 112 6 active sync /dev/sdh >>> 8 8 80 7 active sync /dev/sdf >>> 7 8 64 8 active sync /dev/sde >>> >>> Afterwards I did an fsck: >>> >>> sudo fsck.ext4 -f /dev/md0 >>> e2fsck 1.42.13 (17-May-2015) >>> Pass 1: Checking inodes, blocks, and sizes >>> Pass 2: Checking directory structure >>> Pass 3: Checking directory connectivity >>> Pass 4: Checking reference counts >>> Pass 5: Checking group summary information >>> /dev/md0: 70089/1220923392 files (3.6% non-contiguous), >>> 7726498938/9767368960 blocks >>> >>> Following that, I tried to perform an offline resize: >>> >>> sudo resize2fs /dev/md0 >>> resize2fs 1.42.13 (17-May-2015) >>> Resizing the filesystem on /dev/md0 to 13674316544 (4k) blocks. >>> Should never happen: resize inode corrupt! >>> >>> After having done that, it looks like it should have been an online >>> resize from reading a thread on here from 2015. >>> >>> After trying the resize I tried to do another fsck: >>> >>> sudo fsck.ext4 -f /dev/md0 >>> e2fsck 1.42.13 (17-May-2015) >>> ext2fs_check_desc: Corrupt group descriptor: bad block for inode table >>> fsck.ext4: Group descriptors look bad... trying backup blocks... >>> Superblock has an invalid journal (inode 8). >>> Clear<y>? yes >>> *** ext3 journal has been deleted - filesystem is now ext2 only *** >>> >>> Resize inode not valid. Recreate<y>? yes >>> >>> It's been stuck here for days, with: >>> >>> 14827 root 20 0 141796 121044 2688 R 93.8 0.4 5546:26 fsck.ext4 >>> >>> It's been running at around 100% the whole time. I don't see any disk >>> io happening either. >>> >>> sudo dumpe2fs -h /dev/md0 >>> dumpe2fs 1.42.13 (17-May-2015) >>> Filesystem volume name: <none> >>> Last mounted on: /Media10 >>> Filesystem UUID: d36119d5-e3ec-47f7-b93e-124eb4598367 >>> Filesystem magic number: 0xEF53 >>> Filesystem revision #: 1 (dynamic) >>> Filesystem features: has_journal ext_attr resize_inode dir_index >>> filetype extent 64bit flex_bg sparse_super large_file huge_file >>> uninit_bg dir_nlink extra_isize >>> Filesystem flags: signed_directory_hash >>> Default mount options: user_xattr acl >>> Filesystem state: clean with errors >>> Errors behavior: Continue >>> Filesystem OS type: Linux >>> Inode count: 1709293568 >>> Block count: 13674316544 >>> Reserved block count: 683715825 >>> Free blocks: 5886063280 >>> Free inodes: 1709223479 >>> First block: 0 >>> Block size: 4096 >>> Fragment size: 4096 >>> Group descriptor size: 64 >>> Blocks per group: 32768 >>> Fragments per group: 32768 >>> Inodes per group: 4096 >>> Inode blocks per group: 256 >>> RAID stride: 128 >>> RAID stripe width: 256 >>> Flex block group size: 16 >>> Filesystem created: Sun Dec 17 10:10:08 2017 >>> Last mount time: Sat Mar 9 17:58:06 2019 >>> Last write time: Mon Mar 11 05:48:14 2019 >>> Mount count: 0 >>> Maximum mount count: -1 >>> Last checked: Mon Mar 11 05:16:14 2019 >>> Check interval: 0 (<none>) >>> Lifetime writes: 29 TB >>> Reserved blocks uid: 0 (user root) >>> Reserved blocks gid: 0 (group root) >>> First inode: 11 >>> Inode size: 256 >>> Required extra isize: 28 >>> Desired extra isize: 28 >>> Journal inode: 8 >>> Default directory hash: half_md4 >>> Directory Hash Seed: 23fd4260-aee9-4f36-8406-240f3b7a39d2 >>> Journal backup: inode blocks >>> Journal superblock magic number invalid! >>> >>> >>> Should I let the fsck continue, or is it safe to exit to try something else. >>> >>> I recently did an offline resize a few weeks ago on the same array and >>> it worked out just fine. I'm not sure what happened this time, I >>> followed the same steps. >>> >>> Thanks for any help.
