Hello, I'm writing about poor performances accessing a dm-crypt/LUKS partition using nfs. I was running a fileserver Ubuntu_Server_11.04 (Kernel 2.6.38-13#52) with no problem. But after an upgrade to Ubuntu_Server_11.10 (Kernel 3.0.0-15#26) reading and writing a crypted partition with nfs is very slow. I've made different tests with a 2GB file to show the difference when the server is running a 2.6 or 3.0 kernel. All others parameters are always the same. First, a test using nfs to access a crypted partition (dm-crypt/LUKS/ext4) on the server: *Svr Kernel 3.0.0-15#26 : NFS (R/W) = 30 MB/s / 33 MB/s <-------- The problem is here ! *Svr Kernel 2.6.38-13#52 : NFS (R/W) = 81 MB/s / 53 MB/s As you can see, with the Kernel 3.0, performances are very bad. The flow is always constant at ~30MB; there's clearly a limitation For comparison, the same test, but this time, using nfs to read and write a non-crypted partition (ext4) (same HD of the crypted partition): *Svr Kernel 3.0.0-15#26 : NFS (R/W) = 92 MB/s / 77 MB/s *Svr Kernel 2.6.38-13#52 : NFS (R/W) = 111 MB/s / 74 MB/s Using nfs without dm_crypt, performances are good with the 2 Kernels Is dm_crypt module the problem ? Following, a test without nfs, copying the 2GB file using a 2nd HD on the server from/to the crypted partition: *Svr Kernel 3.0.0-15#26 : cp (from/to) the crypted partition = 64 MB/s / 57 MB/s *Svr Kernel 2.6.38-13#52 : cp (from/to) the crypted partition = 61 MB/s / 60 MB/s Using dm_crypt without nfs, there's no difference between the 2 Kernels Apparently, there is a bad interaction between nfs and dm_crypt only with a 3.0 Kernel ? What king of change has been made in the Kernel 3.x branch ? Do you think that the dm_crypt module is involved ? Or perhaps it's a mapper/buffer problem ? nfs seems to work correctly: no paquets are lost during transfers (I'm using default parameters, except for exportfs: async option instead of sync) Unfortunately, I'm just an user and my knowledge is limited, especially with dm_crypt and device_mapper. Perhaps you can help me with this part ? Feel free to ask more informations / tests. Regards, Mickael Note: - I've build other servers with a debian Wheezy (Kernel 3.1.0-1) and a Fedora 16 (Kernel 3.2.1-3). Both obtained the same results. - Following, hardware/software informations about my system: * Kernel version (from /proc/version): Linux version 3.0.0-15-server (buildd@crested) (gcc version 4.6.1 (Ubuntu/Linaro 4.6.1-9ubuntu3) ) #26-Ubuntu SMP Fri Jan 20 19:07:39 UTC 2012 * Environment (Server) CPU AMD Athlon64 3500+ (1core) 1GB RAM mobo Asus A8N nForce4 eth 1GB nForce4 HD WD 2To Green (sata_nv) :~$ cryptsetup luksDump /dev/sda1 Version: 1 Cipher name: aes Cipher mode: cbc-essiv:sha256 Hash spec: sha1 Payload offset: 2056 MK bits: 256 :~$ mkfs.ext4 /dev/mapper/crypted -m 0.2 :~$ exportfs -v /mnt/crypted 192.168.0.1(rw,async,wdelay,no_root_squash,no_subtree_check) :~$ cat /proc/mount (server) /dev/mapper/crypted /mnt/crypted ext4 rw,relatime,user_xattr,acl,barrier=1,data=ordered 0 0 :~$ cat /proc/mount (client) nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 192.168.0.10:/mnt/crypted/ /mnt/crypted nfs4 rw,relatime,vers=4,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.1,minorversion=0,local_lock=none,addr=192.168.0.10 0 0 :~$ nfsiostat (after reading and writing a 2GB file) 192.168.0.10:/mnt/crypted/ mounted on /mnt/crypted: op/s rpc bklog 1432.16 0.88 read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 32.037 4108.500 128.242 0 (0.0%) 66.974 94.895 write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 47.713 4116.645 86.280 0 (0.0%) 27.833 1814.244 192.168.0.10:/mnt/sda2/ mounted on /mnt/sda2: (sda2 -> ext4 not crypted partition) op/s rpc bklog 9417.74 5.37 read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 199.295 25502.509 127.964 0 (0.0%) 16.920 22.618 write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) 329.141 25555.419 77.643 0 (0.0%) 3.888 507.813 :~$ cat /proc/fs/nfsd export_features: 0x17e3f 0xf exports: /mnt/crypted 192.168.0.1(rw,no_root_squash,async,wdelay,no_subtree_check) max_block_size: 131072 nfsv4gracetime: 90 nfsv4leasetime: 90 nfsv4recoverydir: /var/lib/nfs/v4recovery pool_stats: 0 350070842 335513464 11979013 24 pool_threads: 8 portlist: udp/tcp=2049 supported_krb5_enctypes: 18,17,16,23,3,1,2 threads: 8 versions: +2 +3 +4 +4.1 :~$ cat /sys/kernel/slab/dm_crypt_io/ aliases : [0] align : [8] alloc_calls : [] cache_dma : [0] cpu_slabs : [1 N0=1] ctor : [] destroy_by_rcu : [0] free_calls : [] hwcache_align : [0] min_partial : [7] objects : [26 N0=26] object_size : [152] objects_partial : [0] objs_per_slab : [26] order : [0] partial : [0] poison : [0] reclaim_account : [0] red_zone : [0] remote_node_defrag_ratio : [100] reserved : [0] sanity_checks : [0] shrink : [] slabs : [1 N0=1] slab_size : [152] store_user : [0] total_objects : [26 N0=26] trace : [0] validate : [] _______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
