Here are more tests focusing the CPU load with htop snapshot: with Kernel 3.x, nfsd threads are using more CPU !? Then kworker (process used for encryption) has less CPU time. (The server is a simple fileserver using nfs, dm_crypt and ssh for administration.) About the low non-encrypted performance; this is certainly due to AMD Kool&Quiet. As CPU load is low, the CPU frequency stay at its minimum value 1GHz during non-encrypted access. But when using encryption, the frequency is set to its maximum, 2.3GHz *** KERNEL 2.6 *** NFS Reading: cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi read: 2.032 GB [ 2032330895 B] avg: 81.293 MB/sec [ 25 sec] CPU[||||||||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 36, 68 kthr; 3 running Mem[||||||||||||||||||||||||||||||||||||||||||||81/998MB] Load average: 2.68 2.93 1.79 Swp[ 0/1951MB] Uptime: 00:20:20 PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 41 root 20 0 0 0 0 R 84.0 0.0 2:05.48 kworker/0:1 1020 root 20 0 0 0 0 D 1.0 0.0 0:10.72 nfsd 1015 root 20 0 0 0 0 D 1.0 0.0 0:07.29 nfsd 1019 root 20 0 0 0 0 D 1.0 0.0 0:08.35 nfsd 1022 root 20 0 0 0 0 D 1.0 0.0 0:11.00 nfsd 1016 root 20 0 0 0 0 D 1.0 0.0 0:07.41 nfsd 1017 root 20 0 0 0 0 D 1.0 0.0 0:08.79 nfsd 1021 root 20 0 0 0 0 D 1.0 0.0 0:08.41 nfsd 42 root 20 0 0 0 0 S 1.0 0.0 0:06.89 kswapd0 1018 root 20 0 0 0 0 D 1.0 0.0 0:11.10 nfsd 1460 root 20 0 0 0 0 R 0.0 0.0 0:01.34 kworker/0:0 1475 root 20 0 21972 1556 1148 R 0.0 0.2 0:01.79 htop 1 root 20 0 24188 2088 1168 S 0.0 0.2 0:00.48 /sbin/init ---------- NFS Writing: cat testfile.avi | wcs > /mnt/CRYPT/testfile.avi read: 2.032 GB [ 2032330895 B] avg: 50.808 MB/sec [ 40 sec] CPU[||||||||||||||||||||||||||||||||| 61.8%] Tasks: 36, 68 kthr; 2 running Mem[|||||||||||||||||||||||| 86/998MB] Load average: 3.35 2.32 1.81 Swp[ 0/1951MB] Uptime: 00:27:15 PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 41 root 20 0 0 0 0 S 41.0 0.0 3:21.84 kworker/0:1 1015 root 20 0 0 0 0 S 9.0 0.0 0:09.85 nfsd 1020 root 20 0 0 0 0 S 4.0 0.0 0:15.22 nfsd 1021 root 20 0 0 0 0 S 2.0 0.0 0:11.35 nfsd 1018 root 20 0 0 0 0 R 1.0 0.0 0:15.18 nfsd 1460 root 20 0 0 0 0 S 0.0 0.0 0:02.39 kworker/0:0 1017 root 20 0 0 0 0 S 0.0 0.0 0:12.03 nfsd 1488 root 20 0 0 0 0 S 0.0 0.0 0:01.66 flush-251:3 1016 root 20 0 0 0 0 S 0.0 0.0 0:10.30 nfsd 1442 root 20 0 0 0 0 S 0.0 0.0 0:03.41 jbd2/dm-3-8 42 root 20 0 0 0 0 S 0.0 0.0 0:08.25 kswapd0 1022 root 20 0 0 0 0 S 0.0 0.0 0:14.34 nfsd 1019 root 20 0 0 0 0 S 0.0 0.0 0:11.05 nfsd 91 root 20 0 0 0 0 S 0.0 0.0 0:01.27 kworker/0:2 1 root 20 0 24188 2088 1168 S 0.0 0.2 0:00.48 /sbin/init ======================================== *** KERNEL 3.0 *** NFS Reading: cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi read: 2.032 GB [ 2032330895 B] avg: 30.793 MB/sec [ 66 sec] CPU[||||||||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 36, 65 kthr; 3 running Mem[|||||||||||||||||||||||||||||||| 69/998MB] Load average: 2.02 0.47 0.16 Swp[ 0/1951MB] Uptime: 00:03:02 PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 38 root 20 0 0 0 0 R 36.0 0.0 0:05.17 kworker/0:1 1045 root 20 0 0 0 0 R 36.0 0.0 0:03.07 nfsd 1046 root 20 0 0 0 0 D 3.0 0.0 0:02.23 nfsd 1049 root 20 0 0 0 0 D 3.0 0.0 0:00.52 nfsd 1044 root 20 0 0 0 0 D 3.0 0.0 0:00.70 nfsd 1047 root 20 0 0 0 0 D 3.0 0.0 0:00.54 nfsd 1042 root 20 0 0 0 0 D 3.0 0.0 0:00.51 nfsd 1043 root 20 0 0 0 0 D 3.0 0.0 0:00.49 nfsd 1048 root 20 0 0 0 0 D 3.0 0.0 0:00.50 nfsd 1446 root 20 0 21972 1552 1148 R 0.0 0.2 0:00.68 htop 1 root 20 0 24188 2188 1284 S 0.0 0.2 0:00.46 /sbin/init ---------- NFS Writing: cat testfile.avi | wcs > /mnt/CRYPT/testfile.avi read: 2.032 GB [ 2032330895 B] avg: 36.951 MB/sec [ 55 sec] CPU[||||||||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 36, 61 kthr; 3 running Mem[||||||||||||||||||||||||||||||| 83/998MB] Load average: 2.39 2.26 1.10 Swp[ 0/1951MB] Uptime: 00:07:27 PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 38 root 20 0 0 0 0 R 49.0 0.0 0:52.87 kworker/0:1 1048 root 20 0 0 0 0 R 49.0 0.0 0:12.66 nfsd 1045 root 20 0 0 0 0 D 0.0 0.0 0:16.80 nfsd 1044 root 20 0 0 0 0 D 0.0 0.0 0:09.27 nfsd 1043 root 20 0 0 0 0 D 0.0 0.0 0:10.83 nfsd 1042 root 20 0 0 0 0 D 0.0 0.0 0:08.08 nfsd 1047 root 20 0 0 0 0 D 0.0 0.0 0:09.70 nfsd 1467 root 20 0 21972 1556 1152 R 0.0 0.2 0:00.36 htop 1 root 20 0 24188 2072 1168 S 0.0 0.2 0:00.46 /sbin/init > ----- Mail original ----- > De : Arno Wagner <arno@xxxxxxxxxxx> > À : dm-crypt@xxxxxxxx > Cc : > Envoyé le : Mercredi 8 février 2012 16h26 > Objet : Re: Re : Poor performances with nfs and Kernel 3.x > > > Have you looked at CPU load on the server? Maybe the difference is > just the CPU-power that NFS consumes and decryption is already > CPU-Limited before. The low non-encrypted performance, > compared to local performance, is also a bit fishy. > > Pure speculation, but maybe something is consuming CPU > on the server or the CPU is slowed-down in some way. > If this not a CPU-power issue, then the CPU should have > significant idle percentage during encrypted accesses. > > Arno > > > On Wed, Feb 08, 2012 at 03:04:57PM +0000, Mickael wrote: > > >De?: Arno Wagner <arno@xxxxxxxxxxx> > > >??: dm-crypt@xxxxxxxx > > >Cc?: > > >Envoy? le : Mardi 7 f?vrier 2012 9h33 > > >Objet?: Re: Poor performances with nfs and Kernel 3.x > > > > > >O.k., no such obervation with 3.2.2 on client and server. I got > > > > > >root /gate/tmp>cat ttt | wcs > /dev/null > > >read:? 2.147 GB [? 2147483648 B]? ? avg:? 69.274 MB/sec [? ? 31 sec] > > > > > >(wcs, a.k.a. wc-stream is a small tool I wrote to do real-time > > >monitoring of pipeline throughput and byte count, sources > > >below.) This is with some ramp-up and almost 100% CPU load > > >on the (slower) server. > > > > > >I noticed though that nfsiostat is not the right tool to measure, as > > >it gives you performance over the whole time the device has been > > >mounted, i.e. the throughput number keeps being updated in > > >real-time. > > > > > >Maybe re-run with wcs as shown above (compile instructions are in > > >the header of wcs.c) and also monitor CPU usage on the server > > >while this is rrunning, e.g. with "top". I am not saying you > > >imagine the issue, but lets be sure the measurement is good. > > > > > >If you do not want to compile anything, you could also use > > >something like "time cat ttt > /dev/null" and calculate > > >throughput manually. > > > > > >Arno > > > > > >----- CUTed ----- > > > > > > > > > Hello Arno, > > Here are the new tests with wcs: results are still the same. > > > > /tmp is the tmp directory on the client (7200 rpm HD) > > /CRYPT is a crypted partition on the server (Green HD ~5600 rpm) > > /SAV is a non-crypted partition on the server (Green HD ~5600 rpm) > > CPU monitored with htop > > > > > > *** Server with Kernel 3.0.0-15-server #26 > > > > * From client, using NFS on crypted partition: > > > > cat /tmp/testfile.avi | wcs > /mnt/CRYPT/testfile.avi > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 41.476 MB/sec [??? 49 sec] > > > > cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 29.454 MB/sec [??? 69 sec]??? <--------------- > > > > ----------- > > > > * From client, using NFS on non-crypted partition > > > > cat /tmp/testfile.avi | wcs > /mnt/SAV/testfile.avi > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 72.583 MB/sec [??? 28 sec] > > > > cat /mnt/SAV/testfile.avi | wcs > /tmp/testfile.avi > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 65.559 MB/sec [??? 31 sec] > > > > ----------- > > > > > > * From Server: > > > > cat /mnt/CRYPT/testfile.avi | wcs > /dev/null > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 72.583 MB/sec [??? 28 sec] > > > > cat /mnt/SAV/testfile.avi | wcs > /dev/null > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 67.744 MB/sec [??? 30 sec] > > > > ----------- > > > > cat /mnt/CRYPT/testfile.avi | wcs > /mnt/SAV/testfile.avi > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 56.454 MB/sec [??? 36 sec] > > > > cat /mnt/SAV/testfile.avi | wcs > /mnt/CRYPT/testfile.avi > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 54.928 MB/sec [??? 37 sec] > > > > > > > > =============================== > > > > *** Server with Kernel 2.6.38-13-server #52 > > > > * From client, using NFS on crypted partition: > > > > cat /tmp/testfile.avi | wcs > /mnt/CRYPT/testfile.avi > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 54.928 MB/sec [??? 37 sec]??? CPU=100% > > > > cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 81.293 MB/sec [??? 25 sec]??? CPU=98%??? ??? <------------------- > > > > ----------- > > > > * From client, using NFS on non-crypted partition > > > > cat /tmp/testfile.avi | wcs > /mnt/SAV/testfile.avi > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 63.510 MB/sec [??? 32 sec]??? CPU=50% > > > > cat /mnt/SAV/testfile.avi | wcs > /tmp/testfile.avi > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 63.510 MB/sec [??? 32 sec]??? CPU=25% > > > > > > > > * From Server: > > > > cat /mnt/CRYPT/testfile.avi | wcs > /dev/null > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 84.680 MB/sec [??? 24 sec]??? CPU=98% > > > > cat /mnt/SAV/testfile.avi | wcs > /dev/null > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 67.744 MB/sec [??? 30 sec]??? CPU=25% > > > > ----------- > > > > cat /mnt/CRYPT/testfile.avi | wcs > /mnt/SAV/testfile.avi > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 52.111 MB/sec [??? 39 sec]??? CPU=50%-100% > > > > cat /mnt/SAV/testfile.avi | wcs > /mnt/CRYPT/testfile.avi > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 47.264 MB/sec [??? 43 sec]??? CPU=75%-100% > > _______________________________________________ > > dm-crypt mailing list > > dm-crypt@xxxxxxxx > > http://www.saout.de/mailman/listinfo/dm-crypt > > > > -- > Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@xxxxxxxxxxx > GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F > ---- > One of the painful things about our time is that those who feel certainty > are stupid, and those with any imagination and understanding are filled > with doubt and indecision. -- Bertrand Russell > _______________________________________________ > dm-crypt mailing list > dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt _______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
