Re: ext4 filesystem bad extent error review

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, Jan 03, 2014 at 11:16:02AM +0800, Huang Weller (CM/ESW12-CN) wrote:
> 
> It sounds like the barrier test. We wrote such kind test tool
> before, the test program used ioctl(fd, BLKFLSBUF, 0) to set a
> barrier before next write operation.  Do you think this ioctl is
> enough ? Because I saw the ext4 use it. I will do the test with that
> tool and then let you know the result.

The BLKFLSBUF ioctl does __not__ send a CACHE FLUSH command to the
hardware device.  It forces all of the dirty buffers in memory to the
storage device, and then it invalidates all the buffer cache, but it
does not send a CACHE FLUSH command to the hardware.  Hence, the
hardware is free to write it to its on-disk cache, and not necessarily
guarantee that the data is written to stable store.  (For an example
use case of BLKFLSBUF, we use it in e2fsck to drop the buffer cache
for benchmarking purposes.)

If you want to force a CACHE FLUSH (or barrier, depending on the
underlying transport different names may be given to this operation),
you need to call fsync() on the file descriptor open to the block
device.

> More information about journal block which caused the bad extents
> error: We enabled the mount option journal_checksum in our test.  We
> reproduced the same problem and the journal checksum is correct
> because the journal block will not be replayed if checksum is error.

How did you enable the journal_checksum option?  Note that this is not
safe in general, which is why we don't enable it or the async_commit
mount option by default.  The problem is that currently the journal
replay stops when it hits a bad checksum, and this can leave the file
system in a worse case than it currently is in.  There is a way we
could fix it, by adding per-block checksums to the journal, so we can
skip just the bad block, and then force an efsck afterwards, but that
isn't something we've implemented yet.

That being said, if the journal checksum was valid, and so the
corrupted block was replayed, it does seem to argue against
hardware-induced corruption.

Hmm....  I'm stumped, for the moment.  The journal layer is quite
stable, and we haven't had any problems like this reported in many,
many years.

Let's take this back to first principles.  How reliably can you
reproduce the problem?  How often does it fail?  Is it something where
you can characterize the workload leading to this failure?  Secondly,
is a power drop involved in the reproduction at all, or is this
something that can be reproduced by running some kind of workload, and
then doing a soft reset (i.e., force a kernel reboot, but _not_ do it
via a power drop)?

The other thing to ask is when did this problem first start appearing?
With a kernel upgrade?  A compiler/toolchain upgrade?  Or has it
always been there?  

Regards,

							- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Reiser Filesystem Development]     [Ceph FS]     [Kernel Newbies]     [Security]     [Netfilter]     [Bugtraq]     [Linux FS]     [Yosemite National Park]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Samba]     [Device Mapper]     [Linux Media]

  Powered by Linux