Hello Andrey, > - it does not repro on raid1, raid5 only > - i was unable to repro it w/o filesystem > - it looks like at least stable workaround-fix was found. > > The root of problem are READA requests. Today's stable fix is to return > -EIO for all READA requests in map function. That's interesting. I was already surprised that the cause of the problems were READ requests and not WRITE requests, as those are the easy part of dm-crypt. > That fixes it all. > First of all, there is another rant whether dm-crypt should support > READA bio's at all. For myself I found at least two reasons why it > shouldn't: > 1. it takes a lot of CPU for each request to pass through dm-crypt. it > does not make sense to spend all this cpu time to fill readahead buffer. > on my system there is no noticeable perf difference once I disabled > READA bio's. I had thought about this. I decided to keep them as modern CPUs are very fast and I thought, better waste some CPU cycles to avoid delays by having to move the hard disk head back and forth. > Another issue is why this corruption happens at all and here it looks > very strange. Just a refresher - dm-crypt handles READA bio's same way > as it handles a regular READ - creates a clone (actually a whole new > bio, cloning bi_rw and bi_size, passes it down and decrypts in endio > routine.The corruption looks the following way from inside dm-crypt: > > - fs asks to write 2048 bytes of zeroes to some sector. It always > happens ONLY with requests of 2K and only if data is all zeroes. It > happens on random sectors. Write request goes w/o error. > - someone (i have vague idea how linux cache works) sends a READA bio > with size of 4K for an area starting with the same sector. It goes down > to raid5 and it returns garbage. The data returned is not all zeroes > and it _looks_ like it returns data that was there before zeroes were > written. here goes the corruption. > > If I disable READA in dm-crypt, same pattern with generic READ works fine. Wow. Thanks for figuring this out. Now I finally have a clue on where to start. > Any ideas? Should it be investigated at all? Definitely. Thank you very much for debugging this, I had no clue what was going on there. Disabling READA's seem to be the easy fix, and if there are other good reasons for doing so, we should go ahead. But I'd like to find out what is going on there, perhaps there is some more fundamental problem somewhere that should be fixed.
Attachment:
signature.asc
Description: Dies ist ein digital signierter Nachrichtenteil
