Hi, Well... I have some new and interesting information. After deleting the file, and re-creating from backup, the file system got mounted read only and the following errors were logged: May 8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904376:freeing already freed block (bit 1144) May 8 14:58:15 despair kernel: Aborting journal on device dm-0-8. May 8 14:58:15 despair kernel: EXT4-fs (dm-0): Remounting filesystem read-only May 8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904377:freeing already freed block (bit 1145) May 8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904378:freeing already freed block (bit 1146) May 8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904379:freeing already freed block (bit 1147) May 8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904380:freeing already freed block (bit 1148) May 8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904381:freeing already freed block (bit 1149) May 8 14:58:15 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904382:freeing already freed block (bit 1150) May 8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904383:freeing already freed block (bit 1151) May 8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904384:freeing already freed block (bit 1152) May 8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904385:freeing already freed block (bit 1153) May 8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5124block 167904386:freeing already freed block (bit 1154) May 8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949296:freeing already freed block (bit 13296) May 8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949297:freeing already freed block (bit 13297) May 8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949298:freeing already freed block (bit 13298) May 8 14:58:16 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949299:freeing already freed block (bit 13299) May 8 14:58:17 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949300:freeing already freed block (bit 13300) May 8 14:58:17 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949301:freeing already freed block (bit 13301) May 8 14:58:17 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949302:freeing already freed block (bit 13302) May 8 14:58:17 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949303:freeing already freed block (bit 13303) May 8 14:58:17 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949304:freeing already freed block (bit 13304) May 8 14:58:17 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949305:freeing already freed block (bit 13305) May 8 14:58:17 despair kernel: EXT4-fs error (device dm-0): mb_free_blocks:1411: group 5125block 167949306:freeing already freed block (bit 13306) These are the same blocks slated as multiply claimed :) And then running an FSCK, we got the following: ***** File system check forced at Wed May 8 15:16:50 WEST 2013 ***** fsck 1.41.14 (22-Dec-2010) e2fsck 1.42.3 (14-May-2012) /dev/c/c: recovering journal Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Free blocks count wrong for group #5124 (28170, counted=28159). Fix? yes Free blocks count wrong for group #5125 (25861, counted=25850). Fix? yes Free blocks count wrong (420683133, counted=420644972). Fix? yes Free inodes count wrong (29595347, counted=29595271). Fix? yes /dev/c/c: ***** FILE SYSTEM WAS MODIFIED ***** /dev/c/c: 616825/30212096 files (13.6% non-contiguous), 62748564/483393536 blocks Volume seems clean now after reboot and keeps coming back clean. Any ideas based on this? The old Database was over 1.6 GB in size. The re-created one from backup is much smaller 33MB (as it was compacted before restoring). Once these DB files grow they don't get smaller until compacted. Although I'm sure file size should not have a bearing on this. Many Thanks Stephen Elliott -----Original Message----- From: Theodore Ts'o [mailto:tytso@xxxxxxx] Sent: 03 May 2013 22:32 To: Stephen Elliott Cc: 'Andreas Dilger'; linux-ext4@xxxxxxxxxxxxxxx Subject: Re: 2nd Attempt - FSCK Errors On Fri, May 03, 2013 at 07:42:55PM +0100, Stephen Elliott wrote: > One thing maybe you could explain (and Andreas gave me his take too) > is how you can multiply assigned blocks shared with "0" files. Andreas > offered the suggestion that they may be in the same file. If this were > really the case, I would suspect there to be some file corruption issues etc... It could be that the blocks appear multiple times in the file; or it could be the storage system is returning different data on subsequent reads from the device. What we need is a debugfs dump of the inode on the file system that is having trouble. A debugfs dump of the inode from a completely underlated file system is not useful.... - Ted -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
