Hi Matthew,
I didn't find a list for NVMe, I'll Cc linux-ide in case this is useful
to others.
I just got a Thinkpad P70 skylake laptop and thought I'd try 3 SSDs:
NVMe/M2 vs Sata/M2 vs Sata/6GB.
I'm not a benchmarking filesystem expert, so my apologies if I made some
stupid choices, but hopefully the benchmarks are still useful.
I'll give the conclusions first. The buffer cache issue with NVMe is weird,
maybe there is a non default setting I need to tweak?
a) NVMe is faster for big contiguous I/O, even through dmcrypt
b) M2 NVMe and M2 sata do not seem to go through buffer cache for writes, but SATA/6GB does
c) When using buffer cache, SATA/6GB beats NVMe/M2 and SATA/M2 by a lot on block reads. Why?
d) for random I/O, NVMe is up to 50% faster, and faster with ext4 than btrfs
e) Encrypted NVMe is a bit faster than non encrypted Sata/M2
f) At $328 for 512GB M2/NVMe vs $168 for 500GB M2/Sata, unless you're doing big contiguous I/O, the 2x price difference is hard to justify, cached I/O and random I/O are not significantly faster on NVMe.
3 SSDs:
1) M2/SATA Samsung 850 EVO 500 GB M.2 3.5-Inch SSD (MZ-N5E500BW) 33c/GB
2) M2/NVMe: Samsung 950 PRO -Series 512GB PCIe NVMe - M.2 Internal SSD 2-Inch MZ-V5P512BW 64c/GB
70:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a802 (rev 01)
3) SATA6G: Samsung 850 EVO 2 TB 2.5-Inch SATA III Internal SSD (MZ-75E2T0B/AM) 30c/GB
For price comparison
Seagate 2TB Laptop HDD SATA 6Gb/s 32MB Cache 2.5-Inch Internal Bare Drive (ST2000LM003) 4.6c/GB
Laptop: Skylake Lenovo Thinkpad P70
Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz x 8 (4 cores HT)
Filesystem: btrfs on top of
cryptsetup luksFormat --align-payload=8192 -s 256 -c aes-xts-plain64 /dev/device
ddd test: sync; dd if=/mnt/ram/file of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=100M
dd0 test: sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
Summary:
0) NVMe only really looks faster on big block sequential read/write (indeed 4X read, 2X write).
On Random I/O, it's no faster. Other overheads kill its performance advantage.
1) For Cached I/O NVMe seems slowed down by dmcrypt in hdparm/ioz ranread/write whereas M2/Sata is not
2) 2TB Sata 6GB SSD is faster for read/write encrypted than NVMe is non encrypted (except hdparm)
3) iozone through block cache is faster on M2/SATA than M2/NVMe, weird
4) iozone through block cache is much faster for everything on Sata6G, also weird
5) iozone direct I/O shows NVMe is fastest (by 50% only) and external SATA just a bit faster than M2/Sata
6) when using dmcrypt, NVMe is only marginally faster in iozone than Sata/M2 or Sata/6G
7) kernel build speed is same on SATA/M2 and NVMe/M2, encrypted or not, but SATA6G is 10% faster (probably the same effect that block cache works better on SATA/6G than SATA/M2 or NVMe/M2)
8) ext4 does better on iozone -I (directIO) than btrfs, 2x as fast on random read for NVMe
SATA/M2 dmcrypt SATA/M2 NVMe dmcrypt NVMe SATA6GB/dmcrypt
hdparm -t 534MB/s 517MB/s 1307MB/s 2140MB/s 534MB/s
ddd 10GB read 550MB/s 560MB/s 2000MB/s 2400MB/s 550MB/s
ddd 10GB write 500MB/s 503MB/s 1300MB/s 1200MB/s 506MB/s
dd with /dev/zero hits optimizations that actually penalize NVMe:
dd0 10GB read 7GB/s 6.7GB/s 2.3GB/s 4.4GB/s 6.6GB/s
dd0 10GB write 495MB/s 503MB/s 1.3GB/s 1.3GB/s 2.0GB/s
iozone -e -a -s 4096 -i 0 -i 1 -i 2 (cache makes Sata6G faster than M2, weird):
ioz read 4KB 1760MB/s 1790MB/s 675MB/s 766MB/s 7863MB/s
ioz write 4KB 268MB/s 298MB/s 295MB/s 291MB/s 777MB/s
ioz ranread4KB 5535MB/s 7392MB/s 3185MB/s 7261MB/s 7892MB/s
ioz ranwrite4KB 288MB/s 329MB/s 205MB/s 381MB/s 796MB/s
iozone -I -e -a -s 4096 -i 0 -i 1 -i 2 (direct IO):
SATA/M2 dmcrypt SATA/M2 NVMe dmcrypt NVMe SATA6GB
dmcrypt notcrypted
btrfs btrfs ext4 btrfs btrfs ext4 btrfs ext4
ioz read 4KB 255MB/s 303MB/s 408MB/s 303MB/s 396MB/s 562MB/s 310MB/s 379MB/s
ioz write 4KB 195MB/s 238MB/s 344MB/s 257MB/s 358MB/s 361MB/s 233MB/s 365MB/s
ioz ranread4KB 305MB/s 351MB/s 482MB/s 476MB/s 699MB/s1434MB/s 309MB/s 475MB/s
ioz ranwrite4KB 265MB/s 223MB/s 392MB/s 260MB/s 217MB/s 315MB/s 254MB/s 366MB/s
Details of each test, per type of drive;
================================================================================
saruman M2 SATA 512GB encrypted btrfs
saruman:/tmp# hdparm -t /dev/mapper/cryptroot
Timing buffered disk reads: 1604 MB in 3.00 seconds = 534.03 MB/sec
iozone -e -a -s 4096 -i 0 -i 1 -i 2:
random random
kB reclen write rewrite read reread read write
4096 4 257076 281089 1761711 6094733 6066754 290373
4096 512 277108 295167 1588844 8426437 11128258 318731
4096 4096 268940 308248 1760087 8062616 8377131 320802
iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
4096 4 38626 45227 102236 111141 30936 64317
4096 512 138467 128878 134674 137832 140067 126248
4096 4096 195439 273100 255379 307300 305605 265093
saruman:/mnt/mnt# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
10485760000 bytes (10 GB) copied, 21.1761 s, 495 MB/s
10485760000 bytes (10 GB) copied, 1.49373 s, 7.0 GB/s
kernel 4.4.1 make -j8:
real 17m54.712s
user 126m26.620s
sys 6m21.948s
btrfs send/receive encrypted partition to non encrypted, 87GB: 10mn30
saruman M2 SATA non encrypted btrfs
-----------------------------------
saruman:/mnt/mnt4# hdparm -t /dev/sdc4
/dev/sdc4:
Timing buffered disk reads: 1554 MB in 3.00 seconds = 517.93 MB/sec
iozone -e -a -s 4096 -i 0 -i 1 -i 2:
random random
kB reclen write rewrite read reread read write
4096 4 285020 297565 1739944 7434579 6605595 305785
4096 512 310937 336617 1298278 3703511 5483860 299416
4096 4096 298433 317101 1790911 7392988 7434579 337476
iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
4096 4 47219 93497 118900 126065 39160 86462
4096 512 214381 213388 194307 209215 206951 209289
4096 4096 238792 221839 303768 352951 351342 223618
ext4 iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
4096 4 119524 146035 150173 159105 53090 140697
4096 512 358954 354810 366466 423088 419760 375542
4096 4096 344087 398828 408259 478337 482569 392487
saruman:/mnt/mnt3# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
10485760000 bytes (10 GB) copied, 20.8537 s, 503 MB/s
10485760000 bytes (10 GB) copied, 1.55651 s, 6.7 GB/s
kernel 4.4.1 make -j8:
real 17m55.612s
user 126m31.952s
sys 6m27.452s
********************************************************************************
saruman M2 NVMe 512GB encrypted btrfs
saruman:/tmp# hdparm -t /dev/mapper/cryptroot2
/dev/mapper/cryptroot2:
Timing buffered disk reads: 3924 MB in 3.00 seconds = 1307.56 MB/sec
iozone -e -a -s 4096 -i 0 -i 1 -i 2:
random random
kB reclen write rewrite read reread read write
4096 4 284205 341651 742181 6195843 6178018 341101
4096 512 314351 319573 864883 8865630 8902382 384993
4096 4096 295741 201042 675234 3281854 3185110 205974
iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
4096 4 80042 83160 122616 127129 41421 53963
4096 512 168713 153333 167710 209923 198123 168026
4096 4096 257968 258566 303023 490828 476440 260014
saruman:/mnt/mnt2# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
10485760000 bytes (10 GB) copied, 8.01494 s, 1.3 GB/s
10485760000 bytes (10 GB) copied, 4.63397 s, 2.3 GB/s
kernel 4.4.1 make -j8:
real 17m57.513s
user 126m58.360s
sys 6m25.164s
btrfs send/receive encrypted partition to non encrypted, 87GB: 7mn
(33% faster than Sata M2)
saruman M2 NVMe non encrypted btrfs
-----------------------------------
saruman:/mnt/mnt4# hdparm -t /dev/nvme0n1p4
/dev/nvme0n1p4:
Timing buffered disk reads: 6422 MB in 3.00 seconds = 2140.38 MB/sec
iozone -e -a -s 4096 -i 0 -i 1 -i 2:
random random
kB reclen write rewrite read reread read write
4096 4 274808 271851 684271 2634219 2280673 177630
4096 512 290191 336755 668196 3362795 4039655 184779
4096 4096 291427 340345 766588 7355007 7261741 381768
iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
4096 4 94814 117116 147645 159695 44075 108039
4096 512 271130 267800 239969 325933 301841 271062
4096 4096 358602 275933 396591 719099 699939 217665
ext4 iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
4096 4 107281 200735 205900 263035 54832 124219
4096 512 537478 374641 525060 1173664 1139338 558807
4096 4096 361545 526137 562632 1483477 1434069 315883
saruman:/mnt/mnt4# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
10485760000 bytes (10 GB) copied, 7.90653 s, 1.3 GB/s
10485760000 bytes (10 GB) copied, 2.35863 s, 4.4 GB/s
kernel 4.4.1 make -j8:
real 17m54.221s
user 126m46.264s
sys 6m10.592s
********************************************************************************
saruman Samsung Evo 850 2TB SSD encrypted btrfs
hdparm -t
Timing buffered disk reads: 1606 MB in 3.00 seconds = 534.87 MB/sec
iozone -e -a -s 4096 -i 0 -i 1 -i 2:
random random
kB reclen write rewrite read reread read write
4096 4 513924 738957 8586475 9144037 8017464 697665
4096 512 772100 820362 9287391 10136778 10586522 840551
4096 4096 719099 777517 7863339 8047509 7892238 796108
iozone -I -e -a -s 4096 -i 0 -i 1 -i 2
4096 4 76626 47961 92886 80947 36063 63788
4096 512 135087 138998 143127 150981 144393 132219
4096 4096 233245 234058 310797 311552 309855 254080
ext4 iozone -I -e -a -s 4096 -i 0 -i 1 -i 2 (unencrypted)
4096 4 118013 141212 130575 111534 42343 138654
4096 512 336347 312192 332847 414404 424542 342707
4096 4096 365227 364638 379298 484160 475122 366536
saruman:/tmp# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
10485760000 bytes (10 GB) copied, 5.23919 s, 2.0 GB/s
10485760000 bytes (10 GB) copied, 1.58287 s, 6.6 GB/s
kernel 4.4.1 make -j8:
real 16m4.305s
user 104m0.816s
sys 7m2.832s
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
