Hi qixuan, On 12/05/2012 12:16 AM, qixuan wu wrote: > Hi Tao, all, > > I guess it's a memory(or ext3/kenrel) issue. Beause in one > machine, after report this issue, the partition is made to readonly, > we use debugfs to "ls dir", and it's fine. It can list all files > without error. If the disk has issue, when we using ls command, it > will give error also. (The dir name is also using debugfs to get by > issue inode ID.) Are you sure the disk is good? I just checked the code in e2fsprogs, it seems that it will not complain if rec_len = 0, and the dir iteration just aborts. I guess the right way should be dd the corresponding block out, decode and read it in binary format. :( Thanks Tao > > Is there the possibility: one thread(A) is read_dir(directly read > from buffer head), and another thread(B) is creating item, and fill > this buffer header at the same time. During create item, first modify > the last item's rec_len(let it point to next item which initially is > zero), then fill this added new item. Suppose the seq is below : > 1) B: modify last item's rec_len > 2) A: Read last item, rec_len is modified already by B, and it > identify next item is existing. > 3) A: Read new item, all feilds are zero still. > 3) B: fill new item with correct value. > > This may cause problem. Sorry I am not still checking the code > properly. Raise this suppose is just hope ext3 experts can help to > think whether such concurring scenario has problem or not ? > Any idea or clue is welcome. > > Regards & Thanks a lot. > Wuqx > > On Tue, Dec 4, 2012 at 11:29 PM, Tao Ma <tm@xxxxxx> wrote: >> Hi zefan, >> On 12/04/2012 09:54 PM, Li Zefan wrote: >>>>> We have many x86 boards, and we've been using 2.6.16.60 for a long >>>>> time. Before time we occasionally found ext3 was switched to read-only >>>>> while services were running, and we took it for granted it must be >>>>> some hardware problems. >>>>> But recently this issue happens frequently, both in old boards and >>>>> new boards. We've analyzed logs, and in one board we did find >>>>> exceptional reboot (but ext3 error happened 9 days after), and in >>>>> another board we found mptbase recovery routine, but in all other >>>>> boards there's no suspicious output at all. >>>>> The only change with the system is some application updates, and >>>>> apps now put more IO burden on disks. >>>>> The error always happened in ext3_readdir, like this: >>>>> >>>>> EXT3-fs error (device sda7): ext3_readdir: bad entry in directory#6685458: rec_len is smaller than minimal - offset=3860, inode=0, rec_len=0, name_len=0 >>>>> Aborting journal on device sda7. >>>>> EXT3-fs error (device sda7) in start_transaction: Readonly filesystem >>>>> Aborting journal on device sda7. >>>>> ext3_abort called. >>>>> EXT3-fs error (device sda7): ext3_journal_start_sb: Detected aborted journal >>>>> Remounting filesystem read-only >>>>> __journal_remove_journal_head: freeing b_committed_data >>>>> >>>>> We highly doubt it's hardware failures with this frequency in mind, so >>>>> we're wondering regarding to this issue if there's some ext3 bug-fix >>>>> having merged into mainline but not in our old kernel? >>>> >>>> Absolutely there are. There have been 87 changes just to namei.c since 2.6.16. >>>> You could look through git logs to see if anything looks applicable. >>>> >>>> You might try: >>>> >>>> ef2b02d3e617cb0400eedf2668f86215e1b0e6af ext34: ensure do_split leaves enough free space in both blocks >>> >>> I've been asked to investigate this issue. Thanks for the reply! >>> >>> I found this fix while searching for similar bug reports, but I don't think it >>> worths trying as we don't use dir_index feature. >>> >>> I've collected some logs in different machines, and the error was always >>> triggered in ext3_readdir: >>> >>> EXT3-fs error (device sda7): ext3_readdir: bad entry in directory #6685458: rec_len is smaller than minimal - offset=3860, inode=0, rec_len=0, name_len=0 >>> EXT3-fs error (device sda7): ext3_readdir: bad entry in directory #9650541: rec_len is smaller than minimal - offset=3960, inode=0, rec_len=0, name_len=0 >>> EXT3-fs error (device sda7): ext3_readdir: bad entry in directory #11124783: rec_len is smaller than minimal - offset=4072, inode=0, rec_len=0, name_len=0 >>> EXT3-fs error (device sda7): ext3_readdir: bad entry in directory #52740880: rec_len is smaller than minimal - offset=4024, inode=0, rec_len=0, name_len=0 >>> EXT3-fs error (device sda7): ext3_readdir: bad entry in directory #52740880: rec_len is smaller than minimal - offset=4084, inode=0, rec_len=0, name_len=0 >>> >>> The last two errors happened on the same machine, and the same inode! One >>> happened in 11/22 (I was told they had run fsck later on), and one in 12/01. >> So now this directory has been fscked to be right? You can try by just >> ls this directory and check whether there are any errors in dmesg. >> >> Having said that, as this error happens 2 times for the same inode, >> maybe there is a kernel bug. At least as Ted said in another mail, the >> end of this buffer head seems to be cleared. So I guess next time when >> you see this error, please do: >> 1. use debugfs to find the disk layout for this dir >> 2. read the blocks from the block device directly >> 3. check whether the end of a block(from offset to the end) is zeroed. >> 4. If yes, I guess there should be a kernel bug and we can go on to >> investigate the code. >> >> Thanks >> Tao >>> >>> The offset is always a bit smaller than blocksize, and all the fields are 0. >>> I dumped one of the dirs, and only ~1.6K was used (fsck reported no error). >>> >>> In some machines fsck reported no error at all, and in others filesystems >>> were corrupted though fixable. >>> >>> I didn't see any other error messages before this error at all. >>> >>> Does this remind you of some old ext3 bug? >>> >>> I'll send you fsck output, dir contents and other logs if u'r interested. >>> >>>> >>>> but to be honest, sticking with such an old kernel means you are largely on your own, or may need contract help if you can't resolve it. >>>> >>> >>> There're numerous machines running old kernels, and many of them are hard to >>> change. :( >>> >>> Yesterday they upgrade apps on ~30 machines, and soon after that 5 machines >>> had filesystem corrupted. However they won't stop upgrading other machines! >>> >>> On the other hand, we can hardly reproduce this bug in the lab. >>> >>> So this is critical and urgent. Any help is appreciated. >>> >>> Regards >>> Li Zefan >>> >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >> > -- > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
