On Wed, Feb 08, 2012 at 05:55:21PM +0000, Mickael wrote: > Here are more tests focusing the CPU load with htop snapshot: > > with Kernel 3.x, nfsd threads are using more CPU !? This is what I suspected. You setup is already slightly CPU-limited when reading/writing encrypted partitions. Any additional CPU needed by NFS is then directly deduced from CPU available for encryption. I have to say this is a pretty special case as reading/writing must be CPU limited, but only just about. I have no idea why NFSD needs more CPU though. Maybe add that as finding to your LKML post. Maybe also try to contact the NFS folks directly or via bugzilla.kernel.org by filing a bug against NFSD in 3.x. Possible fixes: 1) live with it 2) stay with kernel 2.6.x 3) use a faster cipher (although AES is pretty optimal here) 4) get a faster CPU, or one with at least 2 cores if this is a single core. Arno > Then kworker (process used for encryption) has less CPU time. > (The server is a simple fileserver using nfs, dm_crypt and ssh for administration.) > > About the low non-encrypted performance; this is certainly due to AMD Kool&Quiet. > As CPU load is low, the CPU frequency stay at its minimum value 1GHz during non-encrypted access. > But when using encryption, the frequency is set to its maximum, 2.3GHz > > > *** KERNEL 2.6 *** > > NFS Reading: > cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 81.293 MB/sec [??? 25 sec] > > ? CPU[||||||||||||||||||||||||||||||||||||||||||||||100.0%]???? Tasks: 36, 68 kthr; 3 running > ? Mem[||||||||||||||||||||||||||||||||||||||||||||81/998MB]???? Load average: 2.68 2.93 1.79 > ? Swp[??????????????????????????????????????????? 0/1951MB]???? Uptime: 00:20:20 > > ? PID USER???? PRI? NI? VIRT?? RES?? SHR S CPU% MEM%?? TIME+? Command > ?? 41 root????? 20?? 0???? 0???? 0???? 0 R 84.0? 0.0? 2:05.48 kworker/0:1 > ?1020 root????? 20?? 0???? 0???? 0???? 0 D? 1.0? 0.0? 0:10.72 nfsd > ?1015 root????? 20?? 0???? 0???? 0???? 0 D? 1.0? 0.0? 0:07.29 nfsd > ?1019 root????? 20?? 0???? 0???? 0???? 0 D? 1.0? 0.0? 0:08.35 nfsd > ?1022 root????? 20?? 0???? 0???? 0???? 0 D? 1.0? 0.0? 0:11.00 nfsd > ?1016 root????? 20?? 0???? 0???? 0???? 0 D? 1.0? 0.0? 0:07.41 nfsd > ?1017 root????? 20?? 0???? 0???? 0???? 0 D? 1.0? 0.0? 0:08.79 nfsd > ?1021 root????? 20?? 0???? 0???? 0???? 0 D? 1.0? 0.0? 0:08.41 nfsd > ?? 42 root????? 20?? 0???? 0???? 0???? 0 S? 1.0? 0.0? 0:06.89 kswapd0 > ?1018 root????? 20?? 0???? 0???? 0???? 0 D? 1.0? 0.0? 0:11.10 nfsd > ?1460 root????? 20?? 0???? 0???? 0???? 0 R? 0.0? 0.0? 0:01.34 kworker/0:0 > ?1475 root????? 20?? 0 21972? 1556? 1148 R? 0.0? 0.2? 0:01.79 htop > ??? 1 root????? 20?? 0 24188? 2088? 1168 S? 0.0? 0.2? 0:00.48 /sbin/init > > > ---------- > NFS Writing: > cat testfile.avi | wcs > /mnt/CRYPT/testfile.avi > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 50.808 MB/sec [??? 40 sec] > > ? CPU[|||||||||||||||||||||||||||||||||????????????? 61.8%]???? Tasks: 36, 68 kthr; 2 running > ? Mem[||||||||||||||||||||||||??????????????????? 86/998MB]???? Load average: 3.35 2.32 1.81 > ? Swp[??????????????????????????????????????????? 0/1951MB]???? Uptime: 00:27:15 > > ? PID USER???? PRI? NI? VIRT?? RES?? SHR S CPU% MEM%?? TIME+? Command > ?? 41 root????? 20?? 0???? 0???? 0???? 0 S 41.0? 0.0? 3:21.84 kworker/0:1 > ?1015 root????? 20?? 0???? 0???? 0???? 0 S? 9.0? 0.0? 0:09.85 nfsd > ?1020 root????? 20?? 0???? 0???? 0???? 0 S? 4.0? 0.0? 0:15.22 nfsd > ?1021 root????? 20?? 0???? 0???? 0???? 0 S? 2.0? 0.0? 0:11.35 nfsd > ?1018 root????? 20?? 0???? 0???? 0???? 0 R? 1.0? 0.0? 0:15.18 nfsd > ?1460 root????? 20?? 0???? 0???? 0???? 0 S? 0.0? 0.0? 0:02.39 kworker/0:0 > ?1017 root????? 20?? 0???? 0???? 0???? 0 S? 0.0? 0.0? 0:12.03 nfsd > ?1488 root????? 20?? 0???? 0???? 0???? 0 S? 0.0? 0.0? 0:01.66 flush-251:3 > ?1016 root????? 20?? 0???? 0???? 0???? 0 S? 0.0? 0.0? 0:10.30 nfsd > ?1442 root????? 20?? 0???? 0???? 0???? 0 S? 0.0? 0.0? 0:03.41 jbd2/dm-3-8 > ?? 42 root????? 20?? 0???? 0???? 0???? 0 S? 0.0? 0.0? 0:08.25 kswapd0 > ?1022 root????? 20?? 0???? 0???? 0???? 0 S? 0.0? 0.0? 0:14.34 nfsd > ?1019 root????? 20?? 0???? 0???? 0???? 0 S? 0.0? 0.0? 0:11.05 nfsd > ?? 91 root????? 20?? 0???? 0???? 0???? 0 S? 0.0? 0.0? 0:01.27 kworker/0:2 > ??? 1 root????? 20?? 0 24188? 2088? 1168 S? 0.0? 0.2? 0:00.48 /sbin/init > > > > ======================================== > > *** KERNEL 3.0 *** > > NFS Reading: > cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 30.793 MB/sec [??? 66 sec] > > ? CPU[||||||||||||||||||||||||||||||||||||||||||||||100.0%]???? Tasks: 36, 65 kthr; 3 running > ? Mem[||||||||||||||||||||||||||||||||??????????? 69/998MB]???? Load average: 2.02 0.47 0.16 > ? Swp[??????????????????????????????????????????? 0/1951MB]???? Uptime: 00:03:02 > > ? PID USER???? PRI? NI? VIRT?? RES?? SHR S CPU% MEM%?? TIME+? Command > ?? 38 root????? 20?? 0???? 0???? 0???? 0 R 36.0? 0.0? 0:05.17 kworker/0:1 > ?1045 root????? 20?? 0???? 0???? 0???? 0 R 36.0? 0.0? 0:03.07 nfsd > ?1046 root????? 20?? 0???? 0???? 0???? 0 D? 3.0? 0.0? 0:02.23 nfsd > ?1049 root????? 20?? 0???? 0???? 0???? 0 D? 3.0? 0.0? 0:00.52 nfsd > ?1044 root????? 20?? 0???? 0???? 0???? 0 D? 3.0? 0.0? 0:00.70 nfsd > ?1047 root????? 20?? 0???? 0???? 0???? 0 D? 3.0? 0.0? 0:00.54 nfsd > ?1042 root????? 20?? 0???? 0???? 0???? 0 D? 3.0? 0.0? 0:00.51 nfsd > ?1043 root????? 20?? 0???? 0???? 0???? 0 D? 3.0? 0.0? 0:00.49 nfsd > ?1048 root????? 20?? 0???? 0???? 0???? 0 D? 3.0? 0.0? 0:00.50 nfsd > ?1446 root????? 20?? 0 21972? 1552? 1148 R? 0.0? 0.2? 0:00.68 htop > ??? 1 root????? 20?? 0 24188? 2188? 1284 S? 0.0? 0.2? 0:00.46 /sbin/init > > > ---------- > NFS Writing: > cat testfile.avi | wcs > /mnt/CRYPT/testfile.avi > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 36.951 MB/sec [??? 55 sec] > > ? CPU[||||||||||||||||||||||||||||||||||||||||||||||100.0%]???? Tasks: 36, 61 kthr; 3 running > ? Mem[|||||||||||||||||||||||||||||||???????????? 83/998MB]???? Load average: 2.39 2.26 1.10 > ? Swp[??????????????????????????????????????????? 0/1951MB]???? Uptime: 00:07:27 > > ? PID USER???? PRI? NI? VIRT?? RES?? SHR S CPU% MEM%?? TIME+? Command > ?? 38 root????? 20?? 0???? 0???? 0???? 0 R 49.0? 0.0? 0:52.87 kworker/0:1 > ?1048 root????? 20?? 0???? 0???? 0???? 0 R 49.0? 0.0? 0:12.66 nfsd > ?1045 root????? 20?? 0???? 0???? 0???? 0 D? 0.0? 0.0? 0:16.80 nfsd > ?1044 root????? 20?? 0???? 0???? 0???? 0 D? 0.0? 0.0? 0:09.27 nfsd > ?1043 root????? 20?? 0???? 0???? 0???? 0 D? 0.0? 0.0? 0:10.83 nfsd > ?1042 root????? 20?? 0???? 0???? 0???? 0 D? 0.0? 0.0? 0:08.08 nfsd > ?1047 root????? 20?? 0???? 0???? 0???? 0 D? 0.0? 0.0? 0:09.70 nfsd > ?1467 root????? 20?? 0 21972? 1556? 1152 R? 0.0? 0.2? 0:00.36 htop > ??? 1 root????? 20?? 0 24188? 2072? 1168 S? 0.0? 0.2? 0:00.46 /sbin/init > > > > > ----- Mail original ----- > > De : Arno Wagner <arno@xxxxxxxxxxx> > > ? : dm-crypt@xxxxxxxx > > Cc : > > Envoy? le : Mercredi 8 f?vrier 2012 16h26 > > Objet : Re: Re : Poor performances with nfs and Kernel 3.x > > > > > > Have you looked at CPU load on the server? Maybe the difference is > > just the CPU-power that NFS consumes and decryption is already > > CPU-Limited before. The low non-encrypted performance, > > compared to local performance, is also a bit fishy. > > > > Pure speculation, but maybe something is consuming CPU > > on the server or the CPU is slowed-down in some way. > > If this not a CPU-power issue, then the CPU should have > > significant idle percentage during encrypted accesses. > > > > Arno > > > > > > On Wed, Feb 08, 2012 at 03:04:57PM +0000, Mickael wrote: > > > >De?: Arno Wagner <arno@xxxxxxxxxxx> > > > >??: dm-crypt@xxxxxxxx > > > >Cc?: > > > >Envoy? le : Mardi 7 f?vrier 2012 9h33 > > > >Objet?: Re: Poor performances with nfs and Kernel 3.x > > > > > > > >O.k., no such obervation with 3.2.2 on client and server. I got > > > > > > > >root /gate/tmp>cat ttt | wcs > /dev/null > > > >read:?? 2.147 GB [?? 2147483648 B]? ? avg:? 69.274 MB/sec [? ? 31 sec] > > > > > > > >(wcs, a.k.a. wc-stream is a small tool I wrote to do real-time > > > >monitoring of pipeline throughput and byte count, sources > > > >below.) This is with some ramp-up and almost 100% CPU load > > > >on the (slower) server. > > > > > > > >I noticed though that nfsiostat is not the right tool to measure, as > > > >it gives you performance over the whole time the device has been > > > >mounted, i.e. the throughput number keeps being updated in > > > >real-time. > > > > > > > >Maybe re-run with wcs as shown above (compile instructions are in > > > >the header of wcs.c) and also monitor CPU usage on the server > > > >while this is rrunning, e.g. with "top". I am not saying you > > > >imagine the issue, but lets be sure the measurement is good. > > > > > > > >If you do not want to compile anything, you could also use > > > >something like "time cat ttt > /dev/null" and calculate > > > >throughput manually. > > > > > > > >Arno > > > > > > > >----- CUTed ----- > > > > > > > > > > > > > Hello Arno, > > > Here are the new tests with wcs: results are still the same. > > > > > > /tmp is the tmp directory on the client (7200 rpm HD) > > > /CRYPT is a crypted partition on the server (Green HD ~5600 rpm) > > > /SAV is a non-crypted partition on the server (Green HD ~5600 rpm) > > > CPU monitored with htop > > > > > > > > > *** Server with Kernel 3.0.0-15-server #26 > > > > > > * From client, using NFS on crypted partition: > > > > > > cat /tmp/testfile.avi | wcs > /mnt/CRYPT/testfile.avi > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 41.476 MB/sec [??? 49 sec] > > > > > > cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 29.454 MB/sec [??? 69 sec]??? <--------------- > > > > > > ----------- > > > > > > * From client, using NFS on non-crypted partition > > > > > > cat /tmp/testfile.avi | wcs > /mnt/SAV/testfile.avi > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 72.583 MB/sec [??? 28 sec] > > > > > > cat /mnt/SAV/testfile.avi | wcs > /tmp/testfile.avi > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 65.559 MB/sec [??? 31 sec] > > > > > > ----------- > > > > > > > > > * From Server: > > > > > > cat /mnt/CRYPT/testfile.avi | wcs > /dev/null > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 72.583 MB/sec [??? 28 sec] > > > > > > cat /mnt/SAV/testfile.avi | wcs > /dev/null > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 67.744 MB/sec [??? 30 sec] > > > > > > ----------- > > > > > > cat /mnt/CRYPT/testfile.avi | wcs > /mnt/SAV/testfile.avi > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 56.454 MB/sec [??? 36 sec] > > > > > > cat /mnt/SAV/testfile.avi | wcs > /mnt/CRYPT/testfile.avi > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 54.928 MB/sec [??? 37 sec] > > > > > > > > > > > > =============================== > > > > > > *** Server with Kernel 2.6.38-13-server #52 > > > > > > * From client, using NFS on crypted partition: > > > > > > cat /tmp/testfile.avi | wcs > /mnt/CRYPT/testfile.avi > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 54.928 MB/sec [??? 37 sec]??? CPU=100% > > > > > > cat /mnt/CRYPT/testfile.avi | wcs > /tmp/testfile.avi > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 81.293 MB/sec [??? 25 sec]??? CPU=98%??? ??? <------------------- > > > > > > ----------- > > > > > > * From client, using NFS on non-crypted partition > > > > > > cat /tmp/testfile.avi | wcs > /mnt/SAV/testfile.avi > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 63.510 MB/sec [??? 32 sec]??? CPU=50% > > > > > > cat /mnt/SAV/testfile.avi | wcs > /tmp/testfile.avi > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 63.510 MB/sec [??? 32 sec]??? CPU=25% > > > > > > > > > > > > * From Server: > > > > > > cat /mnt/CRYPT/testfile.avi | wcs > /dev/null > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 84.680 MB/sec [??? 24 sec]??? CPU=98% > > > > > > cat /mnt/SAV/testfile.avi | wcs > /dev/null > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 67.744 MB/sec [??? 30 sec]??? CPU=25% > > > > > > ----------- > > > > > > cat /mnt/CRYPT/testfile.avi | wcs > /mnt/SAV/testfile.avi > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 52.111 MB/sec [??? 39 sec]??? CPU=50%-100% > > > > > > cat /mnt/SAV/testfile.avi | wcs > /mnt/CRYPT/testfile.avi > > > ?read:?? 2.032 GB [?? 2032330895 B]??? avg:? 47.264 MB/sec [??? 43 sec]??? CPU=75%-100% > > > _______________________________________________ > > > dm-crypt mailing list > > > dm-crypt@xxxxxxxx > > > http://www.saout.de/mailman/listinfo/dm-crypt > > > > > > > -- > > Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@xxxxxxxxxxx > > GnuPG:? ID: 1E25338F? FP: 0C30 5782 9D93 F785 E79C? 0296 797F 6B50 1E25 338F > > ---- > > One of the painful things about our time is that those who feel certainty > > are stupid, and those with any imagination and understanding are filled > > with doubt and indecision. -- Bertrand Russell > > _______________________________________________ > > dm-crypt mailing list > > dm-crypt@xxxxxxxx > http://www.saout.de/mailman/listinfo/dm-crypt > _______________________________________________ > dm-crypt mailing list > dm-crypt@xxxxxxxx > http://www.saout.de/mailman/listinfo/dm-crypt > -- Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@xxxxxxxxxxx GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision. -- Bertrand Russell _______________________________________________ dm-crypt mailing list dm-crypt@xxxxxxxx http://www.saout.de/mailman/listinfo/dm-crypt
