This sounds like one more problems with the Big Kernel Lock. The basic problem (simplified) is that some operations block everything while running. dm-crypt and RAID makes them take longer, but the problem is the blocking. You can possibly provoke the same issue by doing a cat /dev/zero > <some file> on the fileserver, not necessarily to the RAID. As the process priority does not much influence IO usage, that does not really help. As far as I can see, the best current solution is to wait for kernel 2.6.39 (2.6.38 does already help). In the meantime, slowing down rsync i/o (not CPU load) could help. You can try something like this: rsync --bwlimit=1000 ... Gr"usse, Arno On Thu, Apr 21, 2011 at 12:14:48PM -0500, Randall Cotton wrote: > A few months back I brought up a new NFS server that provides user home > directories for my home network. Using Ubuntu 10.04 LTS I set up a 3-disk > RAID 5 device and then dm-crypt on top of that, and finally LVM on top of > that. > > It's been working fine except for one increasingly unacceptable problem. > I've set up a backup system that uses rsync every 5 minutes to take a > snapshot of my home directory and store it on the same disk. This is a > short-term backup solution that lets me get back my work from 5, 10, 15 > minutes ago if I accidentally nuke something. Hard links are used to > minimize the space needed for the snapshots (a la Mike Rubel) > > So this snapshot routine runs on the server itself. The problem is that > when it does run, NFS service essentially goes into grand mal seizure. And > any NFS client accordingly also goes into seizure with the attendant > /var/log/messages entries such as: > > Apr 21 08:52:08 aquamarine kernel: [8073022.344817] nfs: server > pearl.randallcotton.com not responding, still trying > Apr 21 08:52:13 aquamarine kernel: [8073027.401976] nfs: server > pearl.randallcotton.com OK > > And so every 5 minutes, I have to sit in front of my workstation while it > seizes up for 10, 15 even 20 seconds sometimes. As the size of my home > directory grows, the problem gets worse. It's already unacceptable. I > didn't have this problem with my previous NFS server. > > Now my previous NFS server didn't have either a RAID setup, encryption or > LVM, but I'm guessing that kcryptd is the culprit since it's the biggest > CPU hog by far during the seizures. However, cranking down the priority of > kcryptd (and apparently-related processes crypto and kcryptd_io) and also > cranking down the priority of the rsync process and then ALSO cranking *up* > the priority of the nfsd processes seems to help a little, but not much. > > Am I asking too much here (running dm-crypt on an NFS server)? Or does > anyone have an idea how I might adjust things so NFS performance isn't so > hideously cannibalized when kcryptd is busy? > > I don't know dm-crypt internals, so I'm flying a little blind here. If > there's something I should study up on to fix this, let me know. > > Thanks > R > _______________________________________________ > dm-crypt mailing list > dm-crypt@xxxxxxxx > http://www.saout.de/mailman/listinfo/dm-crypt > -- Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@xxxxxxxxxxx GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier _______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
