On Thu, Jan 02, 2014 at 12:59:52PM +0800, Huang Weller (CM/ESW12-CN) wrote: > > We did more test which we backup the journal blocks before we mount the test partition. > Actually, before we mount the test partition, we use fsck.ext4 with -n option to verify whether there is any bad extents issues available. The fsck.ext4 never found any such kind issue. And we can prove that the bad extents issue is happened after journaling replay. Ok, so that implies that the failure is almost certainly due to corrupted blocks in the journal. Hence, when we replay the journal, it causes the the file system to become corrupted, because the "newer" (and presumably, "more correct") metadata blocks found in the blocks recorded in the journal are in fact corrupted. BTW, you can use the logdump command in the debugfs program to look at the journal. The debugfs man page documents it, but once you know the block that was corrupted, which in your case appears to be block 525: debugfs: logdump -b 525 -c Or to see the contents of all of the blocks logged in the journal: debugfs: logdump -ac > > We searched such error on internet, there are some one also has such issue. But there is no solution. > This issue maybe not a big issue which it can be repaired by fsck.ext4 easily. But we have below questions: > 1. whether this issue already been fixed in the latest kernel version? > 2. based on the information I provided in this mail, can you help to solve this issue ? Well, the question is how did the journal get corrupted? It's possible that it's caused by a kernel bug, although I'm not aware of any such bugs being reported. In my mind, the most likely cause is that the SD card is ignoring the CACHE FLUSH command, or is not properly saving the SD card's Flash Translation Layer (FTL) metadata on a power drop. Here are some examples some investigation into lousy SSD's that have this bug --- and historically, SD cards have been **worse** than SSD's, because the manufacturers have a much lower per-unit cost, so they tend to put in even cheaper and crappier FTL systems on SD and eMMC flash. http://lkcl.net/reports/ssd_analysis.html https://www.usenix.org/conference/fast13/understanding-robustness-ssds-under-power-fault What I tell people who are using flash devices is before they start using any flash device, to do power drop testing on a raw device, without any file system present. The simplest way to do this is to write a program that writes consecutive 4k blocks that contain a timestamp, a sequence number, some random data, and a CRC-32 checksum over the contents of the timestamp, sequence number, a flags word, and random data. As the program writes such 4k block, it rolls the dice and once every 64 blocks or so (i.e., pick a random number, and see if it is divisible by 64), then set a bit in the flags word indicating that this block was forced out using a cache flush, and then when writing this block, follow up the write with a CACHE FLUSH command. It's also best if the test program prints the blocks which have been written with CACHE FLUSH to the serial console, and that this is saved by your test rig. (This is what ext4's journal does before and after writing the commit block in the journal, and it guarantees that (a) all of the data in the journal written up to the commit block will be available after a power drop, and (b) that the commit block has been written to the storage device and again, will be available after a power drop.) Once you've written this program, set up a test rig which boots your test board, runs the program, and then drops power to the test board randomly. After the power drop, examine the flash device and make sure that all of the blocks written up to the last "commit block" are in fact valid. You will find that a surprising number of SD cards will fail this test. In fact, the really lousy cards will become unreadable after a power drop. (A fact many wedding photographers discover the hard way they drop their camera and the SD card flies out, and then they find all of that their priceless, once-in-a-lifetime photos are lost forwever.) I ****strongly**** recommend that if you are not testing your SD cards in this way from your parts supplier, you do so immediately, and reject any model that is not able to guarantee that data survives a power drop. Good luck, and I hope this is helpful, - Ted P.S. If you do write such a program, please consider making it available under an open source license. If more companies did this, it would apply pressure to the flash manufacturers to stop making such crappy products, and while it might raise the BOM cost of products by a penny or two, the net result would be better for everyone in the industry. -- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
