ext4 corruption on md [7x1TB in RAID5]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I cleanly shut down my system, and when it came back up, this was
reported:

* Checking file systems...
Sat May  9 13:00:53 2009

fsck 1.41.4 (27-Jan-2009)
fsck.ext4: Unable to resolve
'UUID=52e18bf7-cbe9-4f58-824c-5c943dd074de'^M
fsck died with exit status 8

For some reason it thinks it is ext2:

root@server:/var/log/fsck# fsck -n /dev/md0 | head -5
fsck 1.41.4 (27-Jan-2009)
e2fsck 1.41.4 (27-Jan-2009)
fsck.ext2: Group descriptors look bad... trying backup blocks...
Inode table for group 0 is not in group.  (block 3251545658)
WARNING: SEVERE DATA LOSS POSSIBLE.
Relocate? No



Upon investigation, i'm not sure what to do. I really don't want to lose
the data
on this disk as it will take ages to rebuild (like weeks!).

The system has been in operation for about 2-3months. The data on the
disk
is mostly video + some virtual machines.

Any suggestions for repairing it? I have not done any repairs so far.
Should i just
Run fsck.ext4? Is there something i should try?



root@server:/var/log/fsck# uname -a
Linux server 2.6.28-12-server #43-Ubuntu SMP Fri May 1 20:22:39 UTC 2009
x86_64 GNU/Linux

root@server:/var/log/fsck# fsck.ext4 -n /dev/md0 | head -5
e2fsck 1.41.4 (27-Jan-2009)
fsck.ext4: Group descriptors look bad... trying backup blocks...
Inode table for group 0 is not in group.  (block 3251545658)
WARNING: SEVERE DATA LOSS POSSIBLE.
Relocate? No

root@server:~# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90
  Creation Time : Sun Mar 15 12:15:20 2009
     Raid Level : raid5
     Array Size : 5860574976 (5589.08 GiB 6001.23 GB)
  Used Dev Size : 976762496 (931.51 GiB 1000.20 GB)
   Raid Devices : 7
  Total Devices : 7
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sat May  9 12:42:47 2009
          State : clean
 Active Devices : 7
Working Devices : 7
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : c589dbe9:8dfff933:01f9e43d:ac30fbff (local to host
server)
         Events : 0.305524

    Number   Major   Minor   RaidDevice State
       0       8        0        0      active sync   /dev/sda
       1       8       16        1      active sync   /dev/sdb
       2       8       32        2      active sync   /dev/sdc
       3       8       64        3      active sync   /dev/sde
       4       8       80        4      active sync   /dev/sdf
       5       8       96        5      active sync   /dev/sdg
       6       8      112        6      active sync   /dev/sdh
root@server:~#


root@server:/var/log/fsck# dumpe2fs /dev/md0 |head -60
dumpe2fs 1.41.4 (27-Jan-2009)
ext2fs_read_bb_inode: Invalid argument
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          52e18bf7-cbe9-4f58-824c-5c943dd074de
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype
extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink
extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         not clean with errors
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              366288896
Block count:              1465143744
Reserved block count:     73257187
Free blocks:              1041313557
Free inodes:              366151494
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      674
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Sun Mar 15 12:17:10 2009
Last mount time:          Wed Apr 22 19:04:50 2009
Last write time:          Sat May  9 12:31:43 2009
Mount count:              9
Maximum mount count:      29
Last checked:             Sun Mar 15 12:17:10 2009
Check interval:           15552000 (6 months)
Next check after:         Fri Sep 11 12:17:10 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Default directory hash:   half_md4
Directory Hash Seed:      49785c19-388f-444c-977a-886c147a8ae6
Journal backup:           inode blocks

Group 0: (Blocks 0-32767)
  Checksum 0x70e6, unused inodes 34168
  Primary superblock at 0, Group descriptors at 1-350
  Reserved GDT blocks at 351-1024
  Block bitmap at 98318778 (+98318778), Inode bitmap at 837719297
(+837719297)
  Inode table at 3251545658-3251546169 (+3251545658)
  58662 free blocks, 33159 free inodes, 56553 directories, 34168 unused
inodes
Group 1: (Blocks 32768-65535) [INODE_UNINIT, ITABLE_ZEROED]
  Checksum 0x6275, unused inodes 29973
  Backup superblock at 32768, Group descriptors at 32769-33118
  Reserved GDT blocks at 33119-33792
  Block bitmap at 1123273429 (+1123240661), Inode bitmap at 3031071618
(+3031038850)
  Inode table at 3149300410-3149300921 (+3149267642)
  1881 free blocks, 46059 free inodes, 25281 directories, 29973 unused
inodes
Group 2: (Blocks 65536-98303) [INODE_UNINIT, ITABLE_ZEROED]
  Checksum 0x1a0b, unused inodes 51590
  Block bitmap at 1509085182 (+1509019646), Inode bitmap at 2884607537
(+2884542001)
  Inode table at 1940584034-1940584545 (+1940518498)
  3770 free blocks, 40683 free inodes, 43709 directories, 51590 unused
inodes
Group 3: (Blocks 98304-131071) [ITABLE_ZEROED]
  Checksum 0xc3ec, unused inodes 47423
  Backup superblock at 98304, Group descriptors at 98305-98654
  Reserved GDT blocks at 98655-99328
  Block bitmap at 383505502 (+383407198), Inode bitmap at 1784600367
(+1784502063)
  Inode table at 798930664-798931175 (+798832360)
  63118 free blocks, 7143 free inodes, 19773 directories, 47423 unused
inodes



root@server:/var/log/fsck# dmesg |egrep -i "sd|md|raid|fsck"
[    0.000000]   AMD AuthenticAMD
[    0.000000] RAMDISK: 37778000 - 37fef9b2
[    0.000000] ACPI: RSDP 000FE020, 0014 (r0 INTEL )
[    0.000000] ACPI: RSDT CFEFD038, 004C (r1 INTEL  D975XBX2      AE8
1000013)
[    0.000000] ACPI: DSDT CFEF8000, 3F11 (r1 INTEL  D975XBX2      AE8
MSFT  1000013)
[    0.000000] ACPI: SSDT CFEF2000, 01BC (r1 INTEL     CpuPm      AE8
MSFT  1000013)
[    0.000000] ACPI: SSDT CFEF1000, 0175 (r1 INTEL   Cpu0Ist      AE8
MSFT  1000013)
[    0.000000] ACPI: SSDT CFEF0000, 0175 (r1 INTEL   Cpu1Ist      AE8
MSFT  1000013)
[    0.000000] ACPI: SSDT CFEAB000, 0175 (r1 INTEL   Cpu2Ist      AE8
MSFT  1000013)
[    0.000000] ACPI: SSDT CFEAA000, 0175 (r1 INTEL   Cpu3Ist      AE8
MSFT  1000013)
[    0.000000]   #3 [0037778000 - 0037fef9b2]          RAMDISK ==>
[0037778000 - 0037fef9b2]
[    0.000000]  [ffffe20000000000-ffffe200043fffff] PMD ->
[ffff880028200000-ffff88002c5fffff] on node 0
[    0.013732] ACPI: Checking initramfs for custom DSDT
[    0.920745] ACPI: EC: Look up EC in DSDT
[    1.835081] Fixed MDIO Bus: probed
[    1.835147] Driver 'sd' needs updating - please use bus_type methods
[    1.835891] ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0x40b0
irq 14
[    1.835893] ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x40b8
irq 15
[    2.056689] ata3: SATA max UDMA/133 cmd 0x40c8 ctl 0x40e4 bmdma
0x40a0 irq 19
[    2.056691] ata4: SATA max UDMA/133 cmd 0x40c0 ctl 0x40e0 bmdma
0x40a8 irq 19
[    2.250605] ata3.00: ATA-8: ST31000340AS, SD1A, max UDMA/133
[    2.510628] ata4.00: ATA-8: ST31000340AS, SD1A, max UDMA/133
[    2.591311] scsi 2:0:0:0: Direct-Access     ATA      ST31000340AS
SD1A PQ: 0 ANSI: 5
[    2.591402] sd 2:0:0:0: [sda] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[    2.591414] sd 2:0:0:0: [sda] Write Protect is off
[    2.591416] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    2.591435] sd 2:0:0:0: [sda] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    2.591478] sd 2:0:0:0: [sda] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[    2.591489] sd 2:0:0:0: [sda] Write Protect is off
[    2.591490] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    2.591508] sd 2:0:0:0: [sda] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    2.591511]  sda: unknown partition table
[    2.600717] sd 2:0:0:0: [sda] Attached SCSI disk
[    2.600745] sd 2:0:0:0: Attached scsi generic sg1 type 0
[    2.600855] sd 2:0:1:0: [sdb] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[    2.600866] sd 2:0:1:0: [sdb] Write Protect is off
[    2.600868] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
[    2.600886] sd 2:0:1:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    2.600924] sd 2:0:1:0: [sdb] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[    2.600934] sd 2:0:1:0: [sdb] Write Protect is off
[    2.600936] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
[    2.600954] sd 2:0:1:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    2.600957]  sdb: unknown partition table
[    3.072070] sd 2:0:1:0: [sdb] Attached SCSI disk
[    3.072115] sd 2:0:1:0: Attached scsi generic sg2 type 0
[    3.072178] scsi 3:0:0:0: Direct-Access     ATA      ST31000340AS
SD1A PQ: 0 ANSI: 5
[    3.072237] sd 3:0:0:0: [sdc] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[    3.072248] sd 3:0:0:0: [sdc] Write Protect is off
[    3.072249] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[    3.072268] sd 3:0:0:0: [sdc] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    3.072303] sd 3:0:0:0: [sdc] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[    3.072313] sd 3:0:0:0: [sdc] Write Protect is off
[    3.072315] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[    3.072333] sd 3:0:0:0: [sdc] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    3.072336]  sdc: unknown partition table
[    3.079128] sd 3:0:0:0: [sdc] Attached SCSI disk
[    3.079156] sd 3:0:0:0: Attached scsi generic sg3 type 0
[    3.079257] sd 3:0:1:0: [sdd] 293046768 512-byte hardware sectors:
(150 GB/139 GiB)
[    3.079268] sd 3:0:1:0: [sdd] Write Protect is off
[    3.079269] sd 3:0:1:0: [sdd] Mode Sense: 00 3a 00 00
[    3.079287] sd 3:0:1:0: [sdd] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    3.079322] sd 3:0:1:0: [sdd] 293046768 512-byte hardware sectors:
(150 GB/139 GiB)
[    3.079332] sd 3:0:1:0: [sdd] Write Protect is off
[    3.079334] sd 3:0:1:0: [sdd] Mode Sense: 00 3a 00 00
[    3.079352] sd 3:0:1:0: [sdd] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    3.079354]  sdd: sdd1 sdd2 < sdd5 >
[    3.101054] sd 3:0:1:0: [sdd] Attached SCSI disk
[    3.101107] sd 3:0:1:0: Attached scsi generic sg4 type 0
[    3.470676] ata5.00: ATA-8: ST31000340AS, SD1A, max UDMA/133
[    3.880672] ata6.00: ATA-8: ST31000340AS, SD1A, max UDMA/133
[    4.290698] ata7.00: ATA-8: ST31000340AS, SD1A, max UDMA/133
[    4.700696] ata8.00: ATA-8: ST31000340AS, SD1A, max UDMA/133
[    4.740750] scsi 4:0:0:0: Direct-Access     ATA      ST31000340AS
SD1A PQ: 0 ANSI: 5
[    4.740831] sd 4:0:0:0: [sde] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[    4.740842] sd 4:0:0:0: [sde] Write Protect is off
[    4.740844] sd 4:0:0:0: [sde] Mode Sense: 00 3a 00 00
[    4.740862] sd 4:0:0:0: [sde] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    4.740898] sd 4:0:0:0: [sde] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[    4.740909] sd 4:0:0:0: [sde] Write Protect is off
[    4.740910] sd 4:0:0:0: [sde] Mode Sense: 00 3a 00 00
[    4.740928] sd 4:0:0:0: [sde] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    4.740930]  sde: unknown partition table
[    4.749416] sd 4:0:0:0: [sde] Attached SCSI disk
[    4.749443] sd 4:0:0:0: Attached scsi generic sg5 type 0
[    4.749488] scsi 5:0:0:0: Direct-Access     ATA      ST31000340AS
SD1A PQ: 0 ANSI: 5
[    4.749546] sd 5:0:0:0: [sdf] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[    4.749557] sd 5:0:0:0: [sdf] Write Protect is off
[    4.749559] sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00
[    4.749576] sd 5:0:0:0: [sdf] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    4.749613] sd 5:0:0:0: [sdf] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[    4.749623] sd 5:0:0:0: [sdf] Write Protect is off
[    4.749624] sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00
[    4.749642] sd 5:0:0:0: [sdf] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    4.749645]  sdf: unknown partition table
[    4.759788] sd 5:0:0:0: [sdf] Attached SCSI disk
[    4.759815] sd 5:0:0:0: Attached scsi generic sg6 type 0
[    4.759859] scsi 6:0:0:0: Direct-Access     ATA      ST31000340AS
SD1A PQ: 0 ANSI: 5
[    4.759919] sd 6:0:0:0: [sdg] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[    4.759930] sd 6:0:0:0: [sdg] Write Protect is off
[    4.759932] sd 6:0:0:0: [sdg] Mode Sense: 00 3a 00 00
[    4.759950] sd 6:0:0:0: [sdg] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    4.759986] sd 6:0:0:0: [sdg] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[    4.759997] sd 6:0:0:0: [sdg] Write Protect is off
[    4.759998] sd 6:0:0:0: [sdg] Mode Sense: 00 3a 00 00
[    4.760022] sd 6:0:0:0: [sdg] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    4.760027]  sdg: unknown partition table
[    4.771408] sd 6:0:0:0: [sdg] Attached SCSI disk
[    4.771439] sd 6:0:0:0: Attached scsi generic sg7 type 0
[    4.771483] scsi 7:0:0:0: Direct-Access     ATA      ST31000340AS
SD1A PQ: 0 ANSI: 5
[    4.771540] sd 7:0:0:0: [sdh] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[    4.771551] sd 7:0:0:0: [sdh] Write Protect is off
[    4.771552] sd 7:0:0:0: [sdh] Mode Sense: 00 3a 00 00
[    4.771570] sd 7:0:0:0: [sdh] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    4.771608] sd 7:0:0:0: [sdh] 1953525168 512-byte hardware sectors:
(1.00 TB/931 GiB)
[    4.771618] sd 7:0:0:0: [sdh] Write Protect is off
[    4.771620] sd 7:0:0:0: [sdh] Mode Sense: 00 3a 00 00
[    4.771637] sd 7:0:0:0: [sdh] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    4.771640]  sdh: unknown partition table
[    4.781982] sd 7:0:0:0: [sdh] Attached SCSI disk
[    4.782012] sd 7:0:0:0: Attached scsi generic sg8 type 0
[    4.942482] block sdd5: hash matches
[    5.048354] md: linear personality registered for level -1
[    5.050051] md: multipath personality registered for level -4
[    5.051538] md: raid0 personality registered for level 0
[    5.053697] md: raid1 personality registered for level 1
[    5.271272] raid6: int64x1   1922 MB/s
[    5.441269] raid6: int64x2   2639 MB/s
[    5.611280] raid6: int64x4   2025 MB/s
[    5.781295] raid6: int64x8   1789 MB/s
[    5.951290] raid6: sse2x1    3078 MB/s
[    6.121297] raid6: sse2x2    3651 MB/s
[    6.291275] raid6: sse2x4    7193 MB/s
[    6.291281] raid6: using algorithm sse2x4 (7193 MB/s)
[    6.291283] md: raid6 personality registered for level 6
[    6.291285] md: raid5 personality registered for level 5
[    6.291286] md: raid4 personality registered for level 4
[    6.296306] md: raid10 personality registered for level 10
[    6.494887] md: bind<sde>
[    6.555824] md: bind<sdf>
[    6.622315] md: bind<sdh>
[    6.703948] md: bind<sda>
[    6.780815] md: bind<sdg>
[    7.023507] md: bind<sdc>
[    7.223400] md: bind<sdb>
[    7.226582] raid5: device sdb operational as raid disk 1
[    7.226584] raid5: device sdc operational as raid disk 2
[    7.226586] raid5: device sdg operational as raid disk 5
[    7.226587] raid5: device sda operational as raid disk 0
[    7.226588] raid5: device sdh operational as raid disk 6
[    7.226589] raid5: device sdf operational as raid disk 4
[    7.226591] raid5: device sde operational as raid disk 3
[    7.227201] raid5: allocated 7450kB for md0
[    7.227203] raid5: raid level 5 set md0 active with 7 out of 7
devices, algorithm 2
[    7.227205] RAID5 conf printout:
[    7.227207]  disk 0, o:1, dev:sda
[    7.227208]  disk 1, o:1, dev:sdb
[    7.227209]  disk 2, o:1, dev:sdc
[    7.227210]  disk 3, o:1, dev:sde
[    7.227212]  disk 4, o:1, dev:sdf
[    7.227213]  disk 5, o:1, dev:sdg
[    7.227214]  disk 6, o:1, dev:sdh
[    7.228333]  md0: unknown partition table
[   12.709829] Adding 5992204k swap on /dev/sdd5.  Priority:-1 extents:1
across:5992204k
[   13.245731] EXT3 FS on sdd1, internal journal
[  317.618298] type=1505 audit(1241888758.168:7):
operation="profile_load" name="/usr/sbin/cupsd" name2="default" pid=2550
[  318.016442] Installing knfsd (copyright (C) 1996 okir@xxxxxxxxxxxx).
[  322.767731] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state
recovery directory
[  322.774119] NFSD: starting 90-second grace period

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Index of Archives]     [Reiser Filesystem Development]     [Ceph FS]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Linux FS]     [Yosemite National Park]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Device Mapper]     [Linux Media]

  Powered by Linux