Dear all,
after resizing a 64bit ext4 filesystem from 16TB to 20TB, which
completed without errors, I started a rsync of ~3TB, about 30min into
the copy the following kernel messages started to pop up:
Mar 17 12:03:31 s_local@backup1/backup1 kernel: [2081501.742613] EXT4-fs
(dm-2): mounted filesystem with ordered data mode. Opts: acl,user_xattr
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.489419] EXT4-fs
error (device dm-2): ext4_mb_generate_buddy:755: group 1, 30820 clusters
in bitmap, 30207 in gd
[...]
I attached the full kernel log from the day of the resize.
There are a lot of broken files and when trying to fix them with e2fsck
the program exits:
Inode 2195807 has an invalid extent node (blk 35192760, lblk 0)
Clear? yes
e2fsck: e2fsck_read_bitmaps: illegal bitmap block(s) for /dev/vg-data/backup
/dev/vg-data/backup: ***** FILE SYSTEM WAS MODIFIED *****
e2fsck: aborted
You can find the output of e2fsck -n /dev/vg-data/backup at
http://user.phil.hhu.de/~pranaitis/e2fsck.log.xz
the uncompressed log size is 357MB so I thought I'd rather not send it
via this list.
The kernel version is 3.11-0.bpo.2-amd64 from Debian backports. The
e2fsprogs version is vanilla 1.42.9.
The machine itself is a VM created with VMWare ESXi 5.5.0 using Thick
Provisioned disks.
I don't have any hope of salvaging the filesystem, but is there anything
I can do to help in preventing such issues in the future?
Kind regards,
Jens Pranaitis
--
IKM-Serviceteam der Philosophischen Fakultät
HHU Düsseldorf
Gebäude 23.21, Ebene 04, Raum 88
Tel.: 0211/81-13077
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.701826] scsi 0:0:3:0: Direct-Access VMware Virtual disk 1.0 PQ: 0 ANSI: 2
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.701859] scsi target0:0:3: Beginning Domain Validation
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.702784] scsi target0:0:3: Domain Validation skipping write tests
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.702787] scsi target0:0:3: Ending Domain Validation
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.702851] scsi target0:0:3: FAST-40 WIDE SCSI 80.0 MB/s ST (25 ns, offset 127)
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.703410] sd 0:0:3:0: [sdf] Very big device. Trying to use READ CAPACITY(16).
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.703468] sd 0:0:3:0: [sdf] 8589934592 512-byte logical blocks: (4.39 TB/4.00 TiB)
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.703516] sd 0:0:3:0: [sdf] Write Protect is off
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.703522] sd 0:0:3:0: [sdf] Mode Sense: 61 00 00 00
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.703655] sd 0:0:3:0: [sdf] Cache data unavailable
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.703660] sd 0:0:3:0: [sdf] Assuming drive cache: write through
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.704728] sd 0:0:3:0: Attached scsi generic sg6 type 0
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.704841] sd 0:0:3:0: [sdf] Very big device. Trying to use READ CAPACITY(16).
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.705008] sd 0:0:3:0: [sdf] Cache data unavailable
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.705014] sd 0:0:3:0: [sdf] Assuming drive cache: write through
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.707892] sdf: unknown partition table
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.708146] sd 0:0:3:0: [sdf] Very big device. Trying to use READ CAPACITY(16).
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.708281] sd 0:0:3:0: [sdf] Cache data unavailable
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.708286] sd 0:0:3:0: [sdf] Assuming drive cache: write through
Mar 17 11:08:28 s_local@backup1/backup1 kernel: [2078195.708478] sd 0:0:3:0: [sdf] Attached SCSI disk
Mar 17 11:08:53 s_local@backup1/backup1 kernel: [2078221.071695] sd 0:0:3:0: [sdf] Very big device. Trying to use READ CAPACITY(16).
Mar 17 11:08:53 s_local@backup1/backup1 kernel: [2078221.071849] sd 0:0:3:0: [sdf] Cache data unavailable
Mar 17 11:08:53 s_local@backup1/backup1 kernel: [2078221.071855] sd 0:0:3:0: [sdf] Assuming drive cache: write through
Mar 17 11:08:53 s_local@backup1/backup1 kernel: [2078221.075132] sdf: sdf1
Mar 17 12:03:31 s_local@backup1/backup1 kernel: [2081501.742613] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: acl,user_xattr
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.489419] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 1, 30820 clusters in bitmap, 30207 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.490021] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 2, 30812 clusters in bitmap, 32768 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.490369] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 3, 30628 clusters in bitmap, 30207 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.490675] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 4, 30820 clusters in bitmap, 32768 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.491144] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 5, 30636 clusters in bitmap, 30207 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.491536] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 6, 30628 clusters in bitmap, 32768 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.491882] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 7, 30444 clusters in bitmap, 30207 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.492239] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 8, 30820 clusters in bitmap, 32768 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.492523] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 9, 30620 clusters in bitmap, 30207 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.492698] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 10, 30628 clusters in bitmap, 32768 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.492870] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 11, 30428 clusters in bitmap, 32768 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.493043] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 12, 30636 clusters in bitmap, 32768 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.493215] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 13, 30436 clusters in bitmap, 32768 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.493397] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 14, 30444 clusters in bitmap, 32768 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.494014] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 15, 30244 clusters in bitmap, 32768 in gd
Mar 17 12:37:10 s_local@backup1/backup1 kernel: [2083523.517545] JBD2: Spotted dirty metadata buffer (dev = dm-2, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
Mar 17 12:37:36 s_local@backup1/backup1 kernel: [2083549.233904] EXT4-fs error (device dm-2): mb_free_blocks:1426: group 1607, block 52661289:freeing already freed block (bit 3113)
Mar 17 12:37:36 s_local@backup1/backup1 kernel: [2083549.234045] EXT4-fs error (device dm-2): ext4_mb_generate_buddy:755: group 1607, 32768 clusters in bitmap, 32769 in gd
