On Jun 09, 2008 19:33 +0200, santi@xxxxxxxxxxxx wrote: > That's the scenario: +6TB device on a 3ware 9550SX RAID controller, running > Debian Etch 32bits, with 2.6.25.4 kernel, and defaults e2fsprogs version, > "1.39+1.40-WIP-2006.11.14+dfsg-2etch1". > > Running "tune2fs" returns that filesystem is in EXT3_ERROR_FS state, "clean > with errors": > > # tune2fs -l /dev/sda4 > tune2fs 1.40.10 (21-May-2008) > Filesystem volume name: <none> > Last mounted on: <not available> > Filesystem UUID: 7701b70e-f776-417b-bf31-3693dba56f86 > Filesystem magic number: 0xEF53 > Filesystem revision #: 1 (dynamic) > Filesystem features: has_journal dir_index filetype needs_recovery > sparse_super large_file > Default mount options: (none) > Filesystem state: clean with errors > Errors behavior: Continue > Filesystem OS type: Linux > Inode count: 792576000 > Block count: 1585146848 > > It's a backup storage server, with more than 113 million files, this's the > output of "df -i": > > # df -i /backup/ > Filesystem Inodes IUsed IFree IUse% Mounted on > /dev/sda4 792576000 113385959 679190041 15% /backup > > > Running fsck.ext3 or fsck.ext2 I get: > > # fsck.ext3 /dev/sda4 > e2fsck 1.40.10 (21-May-2008) > Adding dirhash hint to filesystem. > > /dev/sda4 contains a file system with errors, check forced. > Pass 1: Checking inodes, blocks, and sizes I recall that e2fsck allocates on the order of 3 * block_count / 8 bytes, and 5 * inode_count / 8 bytes, so in your case this is about: (5 * 1585146848 + 3 * 792576000) / 8 = 1287932780 bytes = 1.2GB at a minimum, but my estimates might be incorrect. > mmap2(NULL, 99074048, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > 0) = 0x404fa000 Judging by the return values of these functions, this is a 32-bit system, and it is entirely possible that you are exceeding the per-process memory allocation limit. > mmap2(NULL, 748892160, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, > 0) = 0x63be2000 > mmap2(NULL, 1866240000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, > -1, 0) = -1 ENOMEM (Cannot allocate memory) Hmm, it seems a bit excessive to allocate 1.8GB in a single chunk. > Error allocating directory block array: Memory allocation failed > e2fsck: aborted This message is a bit tricky to nail down because it doesn't exist anywhere in the code directly. It is encoded into "e2fsck abbreviations", and the expansion that is normally in the corresponding comment is different. It is PR_1_ALLOCATE_DBCOUNT returned from the call chain: ext2fs_init_dblist-> make_dblist-> ext2fs_get_num_dirs() which is counting the number of directories in the filesystem, and allocating two 12-byte array element for each one. This implies you have 77M directories in your filesystem, or an average of only 10 files per directory? > Appears that fsck is trying to use more than 2GB memory to store inode > table relationship. System has 4GB of physical RAM and 4GB of swap, is > there anyway to limit the memory used by fsck or any solution to check this > filesystem? I don't know offhand how important the dblist structure is, so I'm not sure if there is a way to reduce the memory usage for it. I believe that in low-memory situations it is possible to use tdb in newer versions of e2fsck for the dblist, but I don't know much of the details. > Running fsck with a 64bit LiveCD will solve the problem? Yes, I suspect with a 64-bit kernel you could allocate the full 4GB of RAM for e2fsck and be able to check the filesystem. Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. _______________________________________________ Ext3-users mailing list Ext3-users@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/ext3-users
