Hi again! The very short version is that writing a larger stream of data to an encrypted volume starves all reads on *all* encrypted volumes. Doesn't matter if they reside on different disks or even different controllers. Example: Alice is watching a video (on a windows client using VLC to access the file on a samba share, in case it matters). Then Bob starts backing up his /home of multiple gigs using tar, at which point the video turns into a slideshow. And that's putting it mildly ... Since the unencrypted volumes are fine it seem dm-crypt is the culprit but it might as well be any other layer in our storage setup or, more likely, a combination. Anyway, on the lowest level are a md-raid1 and two -raid5 arrays, grouped into lvm VGs raid1 and raid5 respectively. Some of the LVs on that are encrypted, some aren't. These LVs in turn serve as virtual disks for a handful of kvm boxes (via virtio). OS is Debian 5.02 (lenny) using the 2.6.30 kernel from backports.org, running on an Intel C2Q with 8GB of RAM. I thought 2.6.30 would split the crypto work between the four cores, but it doesn't. Performance maxes out just over 100MiB/s, which is close to the crypto performance of one core. (openssl speed aes-256-cbc gives me ~115MiB/s for one core, ~440MiB/s for all four.) That's probably kvm's fault, since from the pov of the kernel doing the I/O a whole virtual machine is just one process ... and all parallelization, I/O scheduling etc. seems to go out of the window. One of the VMs is our file server and it's doing fine except that writes nearly drown out reads, see above. I've tried messing with the I/O schedulers on the host and guest side, no change. One write stream is enough to bring the whole disk subsystem to its knees. 15 secs to execute /bin/ls as soon as one dd if=/dev/zero of=test bs=4M is running ... All I can think of at this point is that the disks are "too fast" for the CPU. The VM sends down write requests continously, all of which take cycles to encrypt before they even hit the I/O scheduling queue. Now along comes a lonely read that has to wait ages for decryption since the crypt processes are so busy with writes. Since the arrays can write 2-3x faster than the CPU can encrypt, the (I/O scheduling) queue never comes into action ... Any ideas on how I could diagnose the problem are very much appreciated, I'll gladly answer any request for further info. Thanks, Chris _______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
