This sounds more like a general I/O flushing problem. I uses to use a (slow) MO drive for backup and had this all the time after the kernel developers removed easy tuning of the filesystem flush paramaters. (Which I still think was arrogant and stupid.) With earlier flush I did not run into emergency flushes. With the default paramaters I always ran into emergency flushes and the whole machine would lock up for minutes at a time. The lock-up is gone now, likely due to different locking granularity, but I still miss my manually tuned filesystem on occasion, when I run into exactly the starvation you describe. It is possible that tuning the paramaters can be done again by now, I have not looked into it for some time. I think the idea was that early writing has a bit lower performance, but is easier to interrupt. So, possibly no connection to dm-crypt, just the buffer-cache having badly tuned parameters for this type of operation. Arno On Wed, Sep 16, 2009 at 01:54:12AM +0200, Christian Pernegger wrote: > Hi again! > > The very short version is that writing a larger stream of data to an > encrypted volume starves all reads on *all* encrypted volumes. Doesn't > matter if they reside on different disks or even different > controllers. > > Example: Alice is watching a video (on a windows client using VLC to > access the file on a samba share, in case it matters). Then Bob starts > backing up his /home of multiple gigs using tar, at which point the > video turns into a slideshow. And that's putting it mildly ... > > Since the unencrypted volumes are fine it seem dm-crypt is the culprit > but it might as well be any other layer in our storage setup or, more > likely, a combination. > > Anyway, on the lowest level are a md-raid1 and two -raid5 arrays, > grouped into lvm VGs raid1 and raid5 respectively. Some of the LVs on > that are encrypted, some aren't. These LVs in turn serve as virtual > disks for a handful of kvm boxes (via virtio). OS is Debian 5.02 > (lenny) using the 2.6.30 kernel from backports.org, running on an > Intel C2Q with 8GB of RAM. > > I thought 2.6.30 would split the crypto work between the four cores, > but it doesn't. Performance maxes out just over 100MiB/s, which is > close to the crypto performance of one core. (openssl speed > aes-256-cbc gives me ~115MiB/s for one core, ~440MiB/s for all four.) > That's probably kvm's fault, since from the pov of the kernel doing > the I/O a whole virtual machine is just one process ... and all > parallelization, I/O scheduling etc. seems to go out of the window. > > One of the VMs is our file server and it's doing fine except that > writes nearly drown out reads, see above. I've tried messing with the > I/O schedulers on the host and guest side, no change. One write stream > is enough to bring the whole disk subsystem to its knees. 15 secs to > execute /bin/ls as soon as one dd if=/dev/zero of=test bs=4M is > running ... > > All I can think of at this point is that the disks are "too fast" for > the CPU. The VM sends down write requests continously, all of which > take cycles to encrypt before they even hit the I/O scheduling queue. > Now along comes a lonely read that has to wait ages for decryption > since the crypt processes are so busy with writes. Since the arrays > can write 2-3x faster than the CPU can encrypt, the (I/O scheduling) > queue never comes into action ... > > Any ideas on how I could diagnose the problem are very much > appreciated, I'll gladly answer any request for further info. > > Thanks, > > Chris > _______________________________________________ > dm-crypt mailing list > dm-crypt@xxxxxxxx > http://www.saout.de/mailman/listinfo/dm-crypt > -- Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@xxxxxxxxxxx GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier _______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
