On 21.02.2019 15:04, Ondrej Kozina wrote: > Hi Ingo, > > On 2/21/19 1:03 PM, Ingo Franzki wrote: >> Hi, >> >> we just encountered an error when using LVM's pvmove command to move the data from an un-encrypted LVM physical volume onto an encrypted volume. >> After the pvmove has completed, the file system on the logical volume that resides on the moved physical volumes is corrupted. >> >> It seems to be related to a sector size of 4096 used with LUKS2. Once I use the default sector size (512) then the problem does not happen. >> It happens with LUKS2 and even plain mode, as soon as a sector size of 4096 is used. LUKS1 and the default sector size does not show the problem. >> >> Not sure if this is a problem in dm-crypt or LVM, or a combination of both. > > tldr; dm-crypt with encryption sector size set to 4096 may increase device limits to 4096 bytes (minimal i/o size and i/o alignment). It can corrupt filesystem originally created on top of 512B device and moved over to dm-crypt with increased encryption sector size (which you probably did with pvmove). > > ----- > > You're right it's due to encryption sector size, but this is not a bug in dm-crypt nor in pvmove. Let me explain a bit: > > dm-crypt advertise itself as a block device with physical sector size *at least* equal to encryption sector size. Traditionally it's been only 512B. So classical dm-crypt mapped over device with phys. sector size = 512B has no impact. If you mapped dm-crypt over block device with native physical sector size = 4096 you got dm-crypt exposing same limits as underlying block device. Again no problem. Just internally dm-crypt performed encryption on 512B blocks, but it had no impact on exposed limits. > > But things get a bit different with encryption sector size > 512B. > > If you map dm-crypt with encryption sector size set to 4096B over block device with physical sector size = 512B, dm-crypt must increase device limits to 4096. Because when it does encryption it must be aligned to 4096 bytes (and same wrt minimal i/o size). > > So to conclude it: > > If you create dm-crypt with 4K enc. sector over *native* 4K block device it's ok and it must work everytime. > > If you create dm-crypt with 4K enc. sector over 4K block device with 512B emulation, you're ok to create new fs (only new!) on top of it. > > Moving existing filesystem is always a bit tricky. You need to know with what limits the original fs was created. For example on those 4096/512e drives (emulating 512B), fs may have aligned its structures either to 4096B or 512B during mkfs. For example ext4 does the decision based on device size (hope I recall it correctly). I now understand the technical reasons for the behavior, but I still think that this can't be 'works as designed'. You can easily destroy the filesystem and thus loose all your data on your logical volume, without a chance to notice upfront! I really would expect pvmove to issue an error and reject the move when it is about to move between devices with different block sizes, to avoid FS corruption and data loss. At least it should reject the move when it detects that the moved LVs have file systems on it (I assume LVM already knows about this anyway). There probably should be an option to force the move anyway, that I can use when I know what I am doing. But the default behavior can not lead to data loss. > > Just a remark: with LVM2 you may get to similar problems using only lvextend command where you (even accidentally) mix PVs on 512B drives with 4K native drives (latest example: https://bugzilla.redhat.com/show_bug.cgi?id=1669751) Or reject to add PVs with different block sizes at vgextend time already, as mentioned in the bugzilla you mentioned. Either way, it needs to be prevented that using mixed block size PVs lead to data loss! > > (We desperately need to start publishing blog posts related to these issues...sigh) > > Kind regards > O. > > -- Ingo Franzki eMail: ifranzki@xxxxxxxxxxxxx Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/ _______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx https://www.saout.de/mailman/listinfo/dm-crypt
