Re: M2/NVM vs M2/Sata vs SATA/6GB benchmarks

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



For now I've posted this here
http://marc.merlins.org/perso/linux/post_2016-02-22_Thinkpad-P70-NVME-vs-M2-Sata-vs-2_5inch-Sata-SSD.html
along with a few pictures of the thinkpad P70

I'm still not sure why NVMe is slower than Sata when going through the
block cache, but oh well, the rest seems to be working well enough for
now.

Marc

> On Sat, Feb 13, 2016 at 07:13:33PM -0800, Marc MERLIN wrote:
> > Hi Matthew,
> > 
> > I didn't find a list for NVMe, I'll Cc linux-ide in case this is useful
> > to others.
> > 
> > I just got a Thinkpad P70 skylake laptop and thought I'd try 3 SSDs:
> > NVMe/M2 vs Sata/M2 vs Sata/6GB.
> > I'm not a benchmarking filesystem expert, so my apologies if I made some
> > stupid choices, but hopefully the benchmarks are still useful.
> > 
> > I'll give the conclusions first. The buffer cache issue with NVMe is weird,
> > maybe there is a non default setting I need to tweak?
> > 
> > a) NVMe is faster for big contiguous I/O, even through dmcrypt
> > b) M2 NVMe and M2 sata do not seem to go through buffer cache for writes, but SATA/6GB does
> > c) When using buffer cache, SATA/6GB beats NVMe/M2 and SATA/M2 by a lot on block reads. Why?
> > d) for random I/O, NVMe is up to 50% faster, and faster with ext4 than btrfs
> > e) Encrypted NVMe is a bit faster than non encrypted Sata/M2
> > f) At $328 for 512GB M2/NVMe vs $168 for 500GB M2/Sata, unless you're doing big contiguous I/O, the 2x price difference is hard to justify, cached I/O and random I/O are not significantly faster on NVMe.
> > 
> > 
> > 
> > 3 SSDs:
> > 1) M2/SATA Samsung 850 EVO 500 GB M.2 3.5-Inch SSD (MZ-N5E500BW) 33c/GB
> > 
> > 2) M2/NVMe: Samsung 950 PRO -Series 512GB PCIe NVMe - M.2 Internal SSD 2-Inch MZ-V5P512BW 64c/GB
> > 70:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a802 (rev 01)
> > 
> > 3) SATA6G: Samsung 850 EVO 2 TB 2.5-Inch SATA III Internal SSD (MZ-75E2T0B/AM) 30c/GB
> > 
> > For price comparison
> > Seagate 2TB Laptop HDD SATA 6Gb/s 32MB Cache 2.5-Inch Internal Bare Drive (ST2000LM003) 4.6c/GB
> > 
> > Laptop: Skylake Lenovo Thinkpad P70
> > Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz x 8 (4 cores HT)
> > 
> > Filesystem: btrfs on top of 
> > cryptsetup luksFormat --align-payload=8192 -s 256 -c aes-xts-plain64 /dev/device
> > ddd test: sync; dd if=/mnt/ram/file of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=100M
> > dd0 test: sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
> > 
> > Summary: 
> > 0) NVMe only really looks faster on big block sequential read/write (indeed 4X read, 2X write).
> >    On Random I/O, it's no faster. Other overheads kill its performance advantage.
> > 1) For Cached I/O NVMe seems slowed down by dmcrypt in hdparm/ioz ranread/write whereas M2/Sata is not
> > 2) 2TB Sata 6GB SSD is faster for read/write encrypted than NVMe is non encrypted (except hdparm)
> > 3) iozone through block cache is faster on M2/SATA than M2/NVMe, weird
> > 4) iozone through block cache is much faster for everything on Sata6G, also weird
> > 5) iozone direct I/O shows NVMe is fastest (by 50% only) and external SATA just a bit faster than M2/Sata
> > 6) when using dmcrypt, NVMe is only marginally faster in iozone than Sata/M2 or Sata/6G
> > 7) kernel build speed is same on SATA/M2 and NVMe/M2, encrypted or not, but SATA6G is 10% faster (probably the same effect that block cache works better on SATA/6G than SATA/M2 or NVMe/M2)
> > 8) ext4 does better on iozone -I (directIO) than btrfs, 2x as fast on random read for NVMe
> > 
> > 
> > 
> > 		SATA/M2 dmcrypt	SATA/M2		NVMe dmcrypt	NVMe		SATA6GB/dmcrypt
> > hdparm -t	 534MB/s	 517MB/s	1307MB/s	2140MB/s	 534MB/s
> > ddd 10GB read	 550MB/s	 560MB/s	2000MB/s	2400MB/s	 550MB/s
> > ddd 10GB write	 500MB/s	 503MB/s	1300MB/s	1200MB/s	 506MB/s
> > 
> > dd with /dev/zero hits optimizations that actually penalize NVMe:
> > dd0 10GB read	   7GB/s	 6.7GB/s	 2.3GB/s	 4.4GB/s	 6.6GB/s
> > dd0 10GB write	 495MB/s	 503MB/s	 1.3GB/s	 1.3GB/s	 2.0GB/s
> > 
> > iozone -e -a -s 4096 -i 0 -i 1 -i 2 (cache makes Sata6G faster than M2, weird):
> > ioz read 4KB	1760MB/s	1790MB/s	 675MB/s	 766MB/s	7863MB/s
> > ioz write 4KB	 268MB/s	 298MB/s	 295MB/s	 291MB/s	 777MB/s
> > ioz ranread4KB	5535MB/s	7392MB/s	3185MB/s	7261MB/s	7892MB/s
> > ioz ranwrite4KB	 288MB/s	 329MB/s	 205MB/s	 381MB/s	 796MB/s
> > 
> > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2 (direct IO):
> > 		SATA/M2 dmcrypt	 SATA/M2	 NVMe dmcrypt	 NVMe		 SATA6GB
> > 										 dmcrypt notcrypted
> > 		 btrfs		 btrfs   ext4	 btrfs		 btrfs   ext4	 btrfs     ext4
> > ioz read 4KB	 255MB/s	 303MB/s 408MB/s 303MB/s	 396MB/s 562MB/s 310MB/s 379MB/s
> > ioz write 4KB	 195MB/s	 238MB/s 344MB/s 257MB/s	 358MB/s 361MB/s 233MB/s 365MB/s
> > ioz ranread4KB	 305MB/s	 351MB/s 482MB/s 476MB/s	 699MB/s1434MB/s 309MB/s 475MB/s
> > ioz ranwrite4KB	 265MB/s	 223MB/s 392MB/s 260MB/s	 217MB/s 315MB/s 254MB/s 366MB/s
> > 
> > 
> > Details of each test, per type of drive;
> > ================================================================================
> > saruman M2 SATA 512GB encrypted btrfs
> > 
> > saruman:/tmp# hdparm -t /dev/mapper/cryptroot
> >  Timing buffered disk reads: 1604 MB in  3.00 seconds = 534.03 MB/sec
> > 
> > iozone -e -a -s 4096 -i 0 -i 1 -i 2:
> >                                                               random    random
> >               kB  reclen    write  rewrite    read    reread    read     write
> >             4096       4   257076   281089  1761711  6094733  6066754   290373
> >             4096     512   277108   295167  1588844  8426437 11128258   318731
> >             4096    4096   268940   308248  1760087  8062616  8377131   320802
> > 
> > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
> >             4096       4    38626    45227   102236   111141    30936    64317
> >             4096     512   138467   128878   134674   137832   140067   126248
> >             4096    4096   195439   273100   255379   307300   305605   265093
> > 
> > 
> > saruman:/mnt/mnt# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
> > 10485760000 bytes (10 GB) copied, 21.1761 s, 495 MB/s
> > 10485760000 bytes (10 GB) copied, 1.49373 s, 7.0 GB/s
> > 
> > kernel 4.4.1 make -j8:
> > real    17m54.712s
> > user    126m26.620s
> > sys     6m21.948s
> > 
> > btrfs send/receive encrypted partition to non encrypted, 87GB: 10mn30
> > 
> > saruman M2 SATA non encrypted btrfs
> > -----------------------------------
> > saruman:/mnt/mnt4# hdparm -t /dev/sdc4
> > /dev/sdc4:
> >  Timing buffered disk reads: 1554 MB in  3.00 seconds = 517.93 MB/sec
> > 
> > iozone -e -a -s 4096 -i 0 -i 1 -i 2:
> >                                                               random    random
> >               kB  reclen    write  rewrite    read    reread    read     write
> >             4096       4   285020   297565  1739944  7434579  6605595   305785
> >             4096     512   310937   336617  1298278  3703511  5483860   299416
> >             4096    4096   298433   317101  1790911  7392988  7434579   337476
> > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
> >             4096       4    47219    93497   118900   126065    39160    86462
> >             4096     512   214381   213388   194307   209215   206951   209289
> >             4096    4096   238792   221839   303768   352951   351342   223618
> > ext4 iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
> >             4096       4   119524   146035   150173   159105    53090   140697
> >             4096     512   358954   354810   366466   423088   419760   375542
> >             4096    4096   344087   398828   408259   478337   482569   392487
> > 
> > saruman:/mnt/mnt3#  sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
> > 10485760000 bytes (10 GB) copied, 20.8537 s, 503 MB/s
> > 10485760000 bytes (10 GB) copied, 1.55651 s, 6.7 GB/s
> > 
> > kernel 4.4.1 make -j8:
> > real    17m55.612s
> > user    126m31.952s
> > sys     6m27.452s
> > 
> > 
> > ********************************************************************************
> > saruman M2 NVMe 512GB encrypted btrfs
> > 
> > saruman:/tmp# hdparm -t /dev/mapper/cryptroot2
> > /dev/mapper/cryptroot2:
> >  Timing buffered disk reads: 3924 MB in  3.00 seconds = 1307.56 MB/sec
> > 
> > iozone -e -a -s 4096 -i 0 -i 1 -i 2:
> >                                                               random    random
> >               kB  reclen    write  rewrite    read    reread    read     write
> >             4096       4   284205   341651   742181  6195843  6178018   341101
> >             4096     512   314351   319573   864883  8865630  8902382   384993
> >             4096    4096   295741   201042   675234  3281854  3185110   205974
> > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
> >             4096       4    80042    83160   122616   127129    41421    53963
> >             4096     512   168713   153333   167710   209923   198123   168026
> >             4096    4096   257968   258566   303023   490828   476440   260014
> > 
> > 
> > saruman:/mnt/mnt2# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
> > 10485760000 bytes (10 GB) copied, 8.01494 s, 1.3 GB/s
> > 10485760000 bytes (10 GB) copied, 4.63397 s, 2.3 GB/s
> > 
> > kernel 4.4.1 make -j8:
> > real    17m57.513s
> > user    126m58.360s
> > sys     6m25.164s
> > 
> > btrfs send/receive encrypted partition to non encrypted, 87GB: 7mn
> > (33% faster than Sata M2)
> > 
> > saruman M2 NVMe non encrypted btrfs
> > -----------------------------------
> > saruman:/mnt/mnt4# hdparm -t /dev/nvme0n1p4
> > /dev/nvme0n1p4:
> >  Timing buffered disk reads: 6422 MB in  3.00 seconds = 2140.38 MB/sec
> > 
> > 
> > iozone -e -a -s 4096 -i 0 -i 1 -i 2:
> >                                                               random    random
> >               kB  reclen    write  rewrite    read    reread    read     write
> >             4096       4   274808   271851   684271  2634219  2280673   177630
> >             4096     512   290191   336755   668196  3362795  4039655   184779
> >             4096    4096   291427   340345   766588  7355007  7261741   381768
> > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
> >             4096       4    94814   117116   147645   159695    44075   108039
> >             4096     512   271130   267800   239969   325933   301841   271062
> >             4096    4096   358602   275933   396591   719099   699939   217665
> > ext4 iozone -I -e -a -s 4096 -i 0 -i 1 -i 2:
> >             4096       4   107281   200735   205900   263035    54832   124219
> >             4096     512   537478   374641   525060  1173664  1139338   558807
> >             4096    4096   361545   526137   562632  1483477  1434069   315883
> > 
> > saruman:/mnt/mnt4# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
> > 10485760000 bytes (10 GB) copied, 7.90653 s, 1.3 GB/s
> > 10485760000 bytes (10 GB) copied, 2.35863 s, 4.4 GB/s
> > 
> > kernel 4.4.1 make -j8:
> > real    17m54.221s
> > user    126m46.264s
> > sys     6m10.592s
> > 
> > 
> > ********************************************************************************
> > saruman Samsung Evo 850 2TB SSD encrypted btrfs
> > 
> > hdparm -t
> >  Timing buffered disk reads: 1606 MB in  3.00 seconds = 534.87 MB/sec
> > 
> > iozone -e -a -s 4096 -i 0 -i 1 -i 2:
> >                                                               random    random
> >               kB  reclen    write  rewrite    read    reread    read     write
> >             4096       4   513924   738957  8586475  9144037  8017464   697665
> >             4096     512   772100   820362  9287391 10136778 10586522   840551
> >             4096    4096   719099   777517  7863339  8047509  7892238   796108
> > 
> > iozone -I -e -a -s 4096 -i 0 -i 1 -i 2
> >             4096       4    76626    47961    92886    80947    36063    63788
> >             4096     512   135087   138998   143127   150981   144393   132219
> >             4096    4096   233245   234058   310797   311552   309855   254080
> > 
> > ext4 iozone -I -e -a -s 4096 -i 0 -i 1 -i 2 (unencrypted)
> >             4096       4   118013   141212   130575   111534    42343   138654
> >             4096     512   336347   312192   332847   414404   424542   342707
> >             4096    4096   365227   364638   379298   484160   475122   366536
> > 
> > 
> > saruman:/tmp# sync; dd if=/dev/zero of=file bs=100M count=100 conv=fdatasync; dd if=file of=/dev/null bs=1M
> > 10485760000 bytes (10 GB) copied, 5.23919 s, 2.0 GB/s
> > 10485760000 bytes (10 GB) copied, 1.58287 s, 6.6 GB/s
> > 
> > 
> > kernel 4.4.1 make -j8:
> > real    16m4.305s
> > user    104m0.816s
> > sys     7m2.832s
> > 
> > 
> > -- 
> > "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> > Microsoft is to operating systems ....
> >                                       .... what McDonalds is to gourmet cooking
> > Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
> 

-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html



[Index of Archives]     [Linux Filesystems]     [Linux SCSI]     [Linux RAID]     [Git]     [Kernel Newbies]     [Linux Newbie]     [Security]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Samba]     [Device Mapper]

  Powered by Linux