On Mon, Feb 06, 2012 at 10:28:24PM +0000, Mickael wrote: > Hello, > > I'm writing about poor performances accessing a dm-crypt/LUKS partition > using nfs. > > I was running a fileserver Ubuntu_Server_11.04 (Kernel 2.6.38-13#52) with > no problem. But after an upgrade to Ubuntu_Server_11.10 (Kernel > 3.0.0-15#26) reading and writing a crypted partition with nfs is very > slow. Interesting. I think have no issues here with Debian stable and Kernel 3.2.2 on the server and 3.1.6 on the client. What is the kernel version on your client? 3.0.0 is not a very good kernel though with a number of issues. There is also a local privilege excalation in 3.x that has been fixed only in 3.2.2 and later. (Unless Ubuntu did it.) > > I've made different tests with a 2GB file to show the difference when the > server is running a 2.6 or 3.0 kernel. All others parameters are always > the same. > > First, a test using nfs to access a crypted partition (dm-crypt/LUKS/ext4) > on the server: > > *Svr Kernel 3.0.0-15#26? : NFS (R/W) = 30 MB/s / 33 MB/s?? <-------- The problem is here ! > *Svr Kernel 2.6.38-13#52 : NFS (R/W) = 81 MB/s / 53 MB/s > > As you can see, with the Kernel 3.0, performances are very bad. The flow > is always constant at ~30MB; there's clearly a limitation What did you use to do the benchmark? > For comparison, the same test, but this time, using nfs to read and write > a non-crypted partition (ext4) (same HD of the crypted partition): > > *Svr Kernel 3.0.0-15#26? : NFS (R/W) = 92 MB/s / 77 MB/s > *Svr Kernel 2.6.38-13#52 : NFS (R/W) = 111 MB/s / 74 MB/s > Using nfs without dm_crypt, performances are good with the 2 Kernels Interesting. I will do a benchmark here and see whether I have the same. Arno > > Is dm_crypt module the problem ? Following, a test without nfs, copying > the 2GB file using a 2nd HD on the server from/to the crypted partition: > > *Svr Kernel 3.0.0-15#26? : cp (from/to) the crypted partition = 64 MB/s / 57 MB/s > *Svr Kernel 2.6.38-13#52 : cp (from/to) the crypted partition = 61 MB/s / 60 MB/s > Using dm_crypt without nfs, there's no difference between the 2 Kernels > > Apparently, there is a bad interaction between nfs and dm_crypt only with > a 3.0 Kernel ? > > What king of change has been made in the Kernel 3.x branch ? > > Do you think that the dm_crypt module is involved ? Or perhaps it's a > mapper/buffer problem ? nfs seems to work correctly: no paquets are lost > during transfers (I'm using default parameters, except for exportfs: async > option instead of sync) Unfortunately, I'm just an user and my knowledge > is limited, especially with dm_crypt and device_mapper. Perhaps you can > help me with this part ? > > Feel free to ask more informations / tests. > > Regards, > Mickael > > > Note: > - I've build other servers with a debian Wheezy (Kernel 3.1.0-1) and a Fedora 16 (Kernel 3.2.1-3). > Both obtained the same results. > > - Following, hardware/software informations about my system: > > * Kernel version (from /proc/version): > Linux version 3.0.0-15-server (buildd@crested) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #26-Ubuntu SMP Fri Jan 20 19:07:39 UTC 2012 > > > * Environment (Server) > CPU AMD Athlon64 3500+ (1core) > 1GB RAM > mobo Asus A8N nForce4 > eth 1GB nForce4 > HD WD 2To Green (sata_nv) > > > :~$ cryptsetup luksDump /dev/sda1 > Version:?????? ??? 1 > Cipher name:?? ??? aes > Cipher mode:?? ??? cbc-essiv:sha256 > Hash spec:???? ??? sha1 > Payload offset:??? 2056 > MK bits:?????? ??? 256 > > > :~$ mkfs.ext4 /dev/mapper/crypted -m 0.2 > > > :~$ exportfs -v > /mnt/crypted???? ??? 192.168.0.1(rw,async,wdelay,no_root_squash,no_subtree_check) > > > :~$ cat /proc/mount (server) > /dev/mapper/crypted /mnt/crypted ext4 rw,relatime,user_xattr,acl,barrier=1,data=ordered 0 0 > > :~$ cat /proc/mount (client) > nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 > 192.168.0.10:/mnt/crypted/ /mnt/crypted nfs4 rw,relatime,vers=4,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.1,minorversion=0,local_lock=none,addr=192.168.0.10 0 0 > > > :~$ nfsiostat (after reading and writing a 2GB file) > 192.168.0.10:/mnt/crypted/ mounted on /mnt/crypted: > ?? op/s??? ??? rpc bklog > 1432.16 ??? ?? 0.88 > read:???????????? ops/s??? ??? ?? kB/s??? ??? ? kB/op??? ??? retrans??? ??? avg RTT (ms)??? avg exe (ms) > ??? ??? ?32.037 ??? 4108.500 ??? 128.242??????? 0 (0.0%) ??? ?66.974 ??? ?94.895 > write:??????????? ops/s??? ??? ?? kB/s??? ??? ? kB/op??? ??? retrans??? ??? avg RTT (ms)??? avg exe (ms) > ??? ??? ?47.713 ??? 4116.645 ??? ?86.280??????? 0 (0.0%) ??? ?27.833 ??? 1814.244 > > 192.168.0.10:/mnt/sda2/ mounted on /mnt/sda2:? (sda2 -> ext4 not crypted partition) > ?? op/s??? ??? rpc bklog > 9417.74 ??? ?? 5.37 > read:???????????? ops/s??? ??? ?? kB/s??? ??? ? kB/op??? ??? retrans??? ??? avg RTT (ms)??? avg exe (ms) > ??? ??? 199.295 ??? 25502.509 ??? 127.964??????? 0 (0.0%) ??? ?16.920 ??? ?22.618 > write:??????????? ops/s??? ??? ?? kB/s??? ??? ? kB/op??? ??? retrans??? ??? avg RTT (ms)??? avg exe (ms) > ??? ??? 329.141 ??? 25555.419 ??? ?77.643??????? 0 (0.0%) ??? ? 3.888 ??? 507.813 > > > :~$ cat /proc/fs/nfsd > export_features:??? 0x17e3f 0xf > exports:??? /mnt/crypted??? 192.168.0.1(rw,no_root_squash,async,wdelay,no_subtree_check) > max_block_size:??? 131072 > nfsv4gracetime:??? 90 > nfsv4leasetime:??? ??? 90 > nfsv4recoverydir:??? /var/lib/nfs/v4recovery > pool_stats:??? 0 350070842 335513464 11979013 24 > pool_threads:??? 8 > portlist:??? udp/tcp=2049 > supported_krb5_enctypes:??? 18,17,16,23,3,1,2 > threads:??? 8 > versions:??? +2 +3 +4 +4.1 > > > :~$ cat /sys/kernel/slab/dm_crypt_io/ > aliases : [0] > align : [8] > alloc_calls : [] > cache_dma : [0] > cpu_slabs : [1 N0=1] > ctor : [] > destroy_by_rcu : [0] > free_calls : [] > hwcache_align : [0] > min_partial : [7] > objects : [26 N0=26] > object_size : [152] > objects_partial : [0] > objs_per_slab : [26] > order : [0] > partial : [0] > poison : [0] > reclaim_account : [0] > red_zone : [0] > remote_node_defrag_ratio : [100] > reserved : [0] > sanity_checks : [0] > shrink : [] > slabs : [1 N0=1] > slab_size : [152] > store_user : [0] > total_objects : [26 N0=26] > trace : [0] > validate : [] > _______________________________________________ > dm-crypt mailing list > dm-crypt@xxxxxxxx > http://www.saout.de/mailman/listinfo/dm-crypt > -- Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@xxxxxxxxxxx GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision. -- Bertrand Russell _______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
