On Sun, Feb 14, 2016 at 04:32:58AM -0500, Matthew Wilcox wrote: > > Hi Marc, > > I shan't have time to look at this for another week, but I've added > the NVMe mailing list so somebody else can comment in the meantime. Hi Matthew :) > On Sat, Feb 13, 2016 at 07:13:33PM -0800, Marc MERLIN wrote: > > Hi Matthew, > > > > I didn't find a list for NVMe, I'll Cc linux-ide in case this is useful > > to others. > > > > I just got a Thinkpad P70 skylake laptop and thought I'd try 3 SSDs: > > NVMe/M2 vs Sata/M2 vs Sata/6GB. > > I'm not a benchmarking filesystem expert, so my apologies if I made some > > stupid choices, but hopefully the benchmarks are still useful. > > > > I'll give the conclusions first. The buffer cache issue with NVMe is weird, > > maybe there is a non default setting I need to tweak? > > > > a) NVMe is faster for big contiguous I/O, even through dmcrypt > > b) M2 NVMe and M2 sata do not seem to go through buffer cache for writes, but SATA/6GB does > > c) When using buffer cache, SATA/6GB beats NVMe/M2 and SATA/M2 by a lot on block reads. Why? > > d) for random I/O, NVMe is up to 50% faster, and faster with ext4 than btrfs > > e) Encrypted NVMe is a bit faster than non encrypted Sata/M2 > > f) At $328 for 512GB M2/NVMe vs $168 for 500GB M2/Sata, unless you're doing big contiguous I/O, the 2x price difference is hard to justify, cached I/O and random I/O are not significantly faster on NVMe. > > > > > > > > 3 SSDs: > > 1) M2/SATA Samsung 850 EVO 500 GB M.2 3.5-Inch SSD (MZ-N5E500BW) 33c/GB > > > > 2) M2/NVMe: Samsung 950 PRO -Series 512GB PCIe NVMe - M.2 Internal SSD 2-Inch MZ-V5P512BW 64c/GB > > 70:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a802 (rev 01) > > > > 3) SATA6G: Samsung 850 EVO 2 TB 2.5-Inch SATA III Internal SSD (MZ-75E2T0B/AM) 30c/GB > > > > For price comparison > > Seagate 2TB Laptop HDD SATA 6Gb/s 32MB Cache 2.5-Inch Internal Bare Drive (ST2000LM003) 4.6c/GB > > > > Laptop: Skylake Lenovo Thinkpad P70 > > Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz x 8 (4 cores HT) > > > > Filesystem: btrfs on top of > > cryptsetup luksFormat --align-payload=8192 -s 256 -c aes-xts-plain64 /dev/device > > ddd test: sync; dd if=/mnt/ram/file of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=100M > > dd0 test: sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M > > > > Summary: > > 0) NVMe only really looks faster on big block sequential read/write (indeed 4X read, 2X write). > > On Random I/O, it's no faster. Other overheads kill its performance advantage. > > 1) For Cached I/O NVMe seems slowed down by dmcrypt in hdparm/ioz ranread/write whereas M2/Sata is not > > 2) 2TB Sata 6GB SSD is faster for read/write encrypted than NVMe is non encrypted (except hdparm) > > 3) iozone through block cache is faster on M2/SATA than M2/NVMe, weird > > 4) iozone through block cache is much faster for everything on Sata6G, also weird > > 5) iozone direct I/O shows NVMe is fastest (by 50% only) and external SATA just a bit faster than M2/Sata > > 6) when using dmcrypt, NVMe is only marginally faster in iozone than Sata/M2 or Sata/6G > > 7) kernel build speed is same on SATA/M2 and NVMe/M2, encrypted or not, but SATA6G is 10% faster (probably the same effect that block cache works better on SATA/6G than SATA/M2 or NVMe/M2) > > 8) ext4 does better on iozone -I (directIO) than btrfs, 2x as fast on random read for NVMe > > > > > > > > SATA/M2 dmcrypt SATA/M2 NVMe dmcrypt NVMe SATA6GB/dmcrypt > > hdparm -t 534MB/s 517MB/s 1307MB/s 2140MB/s 534MB/s > > ddd 10GB read 550MB/s 560MB/s 2000MB/s 2400MB/s 550MB/s > > ddd 10GB write 500MB/s 503MB/s 1300MB/s 1200MB/s 506MB/s > > > > dd with /dev/zero hits optimizations that actually penalize NVMe: > > dd0 10GB read 7GB/s 6.7GB/s 2.3GB/s 4.4GB/s 6.6GB/s > > dd0 10GB write 495MB/s 503MB/s 1.3GB/s 1.3GB/s 2.0GB/s > > > > iozone -e -a -s 4096 -i 0 -i 1 -i 2 (cache makes Sata6G faster than M2, weird): > > ioz read 4KB 1760MB/s 1790MB/s 675MB/s 766MB/s 7863MB/s > > ioz write 4KB 268MB/s 298MB/s 295MB/s 291MB/s 777MB/s > > ioz ranread4KB 5535MB/s 7392MB/s 3185MB/s 7261MB/s 7892MB/s > > ioz ranwrite4KB 288MB/s 329MB/s 205MB/s 381MB/s 796MB/s > > > > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2 (direct IO): > > SATA/M2 dmcrypt SATA/M2 NVMe dmcrypt NVMe SATA6GB > > dmcrypt notcrypted > > btrfs btrfs ext4 btrfs btrfs ext4 btrfs ext4 > > ioz read 4KB 255MB/s 303MB/s 408MB/s 303MB/s 396MB/s 562MB/s 310MB/s 379MB/s > > ioz write 4KB 195MB/s 238MB/s 344MB/s 257MB/s 358MB/s 361MB/s 233MB/s 365MB/s > > ioz ranread4KB 305MB/s 351MB/s 482MB/s 476MB/s 699MB/s1434MB/s 309MB/s 475MB/s > > ioz ranwrite4KB 265MB/s 223MB/s 392MB/s 260MB/s 217MB/s 315MB/s 254MB/s 366MB/s > > > > > > Details of each test, per type of drive; > > ================================================================================ > > saruman M2 SATA 512GB encrypted btrfs > > > > saruman:/tmp# hdparm -t /dev/mapper/cryptroot > > Timing buffered disk reads: 1604 MB in 3.00 seconds = 534.03 MB/sec > > > > iozone -e -a -s 4096 -i 0 -i 1 -i 2: > > random random > > kB reclen write rewrite read reread read write > > 4096 4 257076 281089 1761711 6094733 6066754 290373 > > 4096 512 277108 295167 1588844 8426437 11128258 318731 > > 4096 4096 268940 308248 1760087 8062616 8377131 320802 > > > > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2: > > 4096 4 38626 45227 102236 111141 30936 64317 > > 4096 512 138467 128878 134674 137832 140067 126248 > > 4096 4096 195439 273100 255379 307300 305605 265093 > > > > > > saruman:/mnt/mnt# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M > > 10485760000 bytes (10 GB) copied, 21.1761 s, 495 MB/s > > 10485760000 bytes (10 GB) copied, 1.49373 s, 7.0 GB/s > > > > kernel 4.4.1 make -j8: > > real 17m54.712s > > user 126m26.620s > > sys 6m21.948s > > > > btrfs send/receive encrypted partition to non encrypted, 87GB: 10mn30 > > > > saruman M2 SATA non encrypted btrfs > > ----------------------------------- > > saruman:/mnt/mnt4# hdparm -t /dev/sdc4 > > /dev/sdc4: > > Timing buffered disk reads: 1554 MB in 3.00 seconds = 517.93 MB/sec > > > > iozone -e -a -s 4096 -i 0 -i 1 -i 2: > > random random > > kB reclen write rewrite read reread read write > > 4096 4 285020 297565 1739944 7434579 6605595 305785 > > 4096 512 310937 336617 1298278 3703511 5483860 299416 > > 4096 4096 298433 317101 1790911 7392988 7434579 337476 > > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2: > > 4096 4 47219 93497 118900 126065 39160 86462 > > 4096 512 214381 213388 194307 209215 206951 209289 > > 4096 4096 238792 221839 303768 352951 351342 223618 > > ext4 iozone -I -e -a -s 4096 -i 0 -i 1 -i 2: > > 4096 4 119524 146035 150173 159105 53090 140697 > > 4096 512 358954 354810 366466 423088 419760 375542 > > 4096 4096 344087 398828 408259 478337 482569 392487 > > > > saruman:/mnt/mnt3# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M > > 10485760000 bytes (10 GB) copied, 20.8537 s, 503 MB/s > > 10485760000 bytes (10 GB) copied, 1.55651 s, 6.7 GB/s > > > > kernel 4.4.1 make -j8: > > real 17m55.612s > > user 126m31.952s > > sys 6m27.452s > > > > > > ******************************************************************************** > > saruman M2 NVMe 512GB encrypted btrfs > > > > saruman:/tmp# hdparm -t /dev/mapper/cryptroot2 > > /dev/mapper/cryptroot2: > > Timing buffered disk reads: 3924 MB in 3.00 seconds = 1307.56 MB/sec > > > > iozone -e -a -s 4096 -i 0 -i 1 -i 2: > > random random > > kB reclen write rewrite read reread read write > > 4096 4 284205 341651 742181 6195843 6178018 341101 > > 4096 512 314351 319573 864883 8865630 8902382 384993 > > 4096 4096 295741 201042 675234 3281854 3185110 205974 > > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2: > > 4096 4 80042 83160 122616 127129 41421 53963 > > 4096 512 168713 153333 167710 209923 198123 168026 > > 4096 4096 257968 258566 303023 490828 476440 260014 > > > > > > saruman:/mnt/mnt2# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M > > 10485760000 bytes (10 GB) copied, 8.01494 s, 1.3 GB/s > > 10485760000 bytes (10 GB) copied, 4.63397 s, 2.3 GB/s > > > > kernel 4.4.1 make -j8: > > real 17m57.513s > > user 126m58.360s > > sys 6m25.164s > > > > btrfs send/receive encrypted partition to non encrypted, 87GB: 7mn > > (33% faster than Sata M2) > > > > saruman M2 NVMe non encrypted btrfs > > ----------------------------------- > > saruman:/mnt/mnt4# hdparm -t /dev/nvme0n1p4 > > /dev/nvme0n1p4: > > Timing buffered disk reads: 6422 MB in 3.00 seconds = 2140.38 MB/sec > > > > > > iozone -e -a -s 4096 -i 0 -i 1 -i 2: > > random random > > kB reclen write rewrite read reread read write > > 4096 4 274808 271851 684271 2634219 2280673 177630 > > 4096 512 290191 336755 668196 3362795 4039655 184779 > > 4096 4096 291427 340345 766588 7355007 7261741 381768 > > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2: > > 4096 4 94814 117116 147645 159695 44075 108039 > > 4096 512 271130 267800 239969 325933 301841 271062 > > 4096 4096 358602 275933 396591 719099 699939 217665 > > ext4 iozone -I -e -a -s 4096 -i 0 -i 1 -i 2: > > 4096 4 107281 200735 205900 263035 54832 124219 > > 4096 512 537478 374641 525060 1173664 1139338 558807 > > 4096 4096 361545 526137 562632 1483477 1434069 315883 > > > > saruman:/mnt/mnt4# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M > > 10485760000 bytes (10 GB) copied, 7.90653 s, 1.3 GB/s > > 10485760000 bytes (10 GB) copied, 2.35863 s, 4.4 GB/s > > > > kernel 4.4.1 make -j8: > > real 17m54.221s > > user 126m46.264s > > sys 6m10.592s > > > > > > ******************************************************************************** > > saruman Samsung Evo 850 2TB SSD encrypted btrfs > > > > hdparm -t > > Timing buffered disk reads: 1606 MB in 3.00 seconds = 534.87 MB/sec > > > > iozone -e -a -s 4096 -i 0 -i 1 -i 2: > > random random > > kB reclen write rewrite read reread read write > > 4096 4 513924 738957 8586475 9144037 8017464 697665 > > 4096 512 772100 820362 9287391 10136778 10586522 840551 > > 4096 4096 719099 777517 7863339 8047509 7892238 796108 > > > > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2 > > 4096 4 76626 47961 92886 80947 36063 63788 > > 4096 512 135087 138998 143127 150981 144393 132219 > > 4096 4096 233245 234058 310797 311552 309855 254080 > > > > ext4 iozone -I -e -a -s 4096 -i 0 -i 1 -i 2 (unencrypted) > > 4096 4 118013 141212 130575 111534 42343 138654 > > 4096 512 336347 312192 332847 414404 424542 342707 > > 4096 4096 365227 364638 379298 484160 475122 366536 > > > > > > saruman:/tmp# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M > > 10485760000 bytes (10 GB) copied, 5.23919 s, 2.0 GB/s > > 10485760000 bytes (10 GB) copied, 1.58287 s, 6.6 GB/s > > > > > > kernel 4.4.1 make -j8: > > real 16m4.305s > > user 104m0.816s > > sys 7m2.832s > > > > > > -- > > "A mouse is a device used to point at the xterm you want to type in" - A.S.R. > > Microsoft is to operating systems .... > > .... what McDonalds is to gourmet cooking > > Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 > -- "A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems .... .... what McDonalds is to gourmet cooking Home page: http://marc.merlins.org/ | PGP 1024R/763BE901 -- To unsubscribe from this list: send the line "unsubscribe linux-ide" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
