Samsung PM883 3.84TB SSD performance

[mj <lists@xxxxxxxxxxxxx>]

Hi,

We purchased Samsung PM883 3.84TB SSDs to be used as BlueStore SSDs in 
our cluster.

I ran some benchmarks (write cache disabled, and with -sync=1 and 
-fsync=1) but with my little understandig, the results look terribly 
bad. (iops around 82!)

Reading pages like yourcmc.ru, we should at least aim for > 1000s IOPS. 
(like 10000 up to 20000)

Here is our complete output:

> root@pve:~# hdparm -W 0 /dev/sdd
>
> /dev/sdd:
>  setting drive write-caching to 0 (off)
>  write-caching =  0 (off)
> root@pve:~# fio -ioengine=libaio -fsync=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd
> test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=32
> fio-2.16
> Starting 1 process
> Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s]     
> test: (groupid=0, jobs=1): err= 0: pid=3268778: Tue Jan 19 14:13:44 2021
>   write: io=19728MB, bw=336674KB/s, iops=82, runt= 60003msec
>     slat (usec): min=1853, max=4652, avg=2428.94, stdev=361.92
>     clat (msec): min=9, max=652, avg=376.87, stdev=22.88
>      lat (msec): min=11, max=655, avg=379.30, stdev=22.94
>     clat percentiles (msec):
>      |  1.00th=[  363],  5.00th=[  367], 10.00th=[  371], 20.00th=[  371],
>      | 30.00th=[  371], 40.00th=[  375], 50.00th=[  375], 60.00th=[  379],
>      | 70.00th=[  379], 80.00th=[  383], 90.00th=[  388], 95.00th=[  392],
>      | 99.00th=[  404], 99.50th=[  420], 99.90th=[  611], 99.95th=[  635],
>      | 99.99th=[  652]
>     lat (msec) : 10=0.02%, 50=0.06%, 100=0.10%, 250=0.24%, 500=99.25%
>     lat (msec) : 750=0.32%
>   cpu          : usr=2.95%, sys=28.30%, ctx=47835, majf=0, minf=26
>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, >=64=0.0%
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
>      issued    : total=r=0/w=4932/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
>      latency   : target=0, window=0, percentile=100.00%, depth=32
>
> Run status group 0 (all jobs):
>   WRITE: io=19728MB, aggrb=336674KB/s, minb=336674KB/s, maxb=336674KB/s, mint=60003msec, maxt=60003msec
>
> Disk stats (read/write):
>   sdd: ios=41/39330, merge=0/4995672, ticks=8/231816, in_queue=231440, util=79.43%
> root@pve:~# fio -ioengine=libaio -sync=1 -invalidate=1 -name=test -bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd
> test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=32
> fio-2.16
> Starting 1 process
> Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s]     
> test: (groupid=0, jobs=1): err= 0: pid=3270282: Tue Jan 19 14:14:58 2021
>   write: io=19468MB, bw=332248KB/s, iops=81, runt= 60001msec
>     slat (usec): min=11658, max=14287, avg=12318.59, stdev=371.39
>     clat (usec): min=8, max=400078, avg=380931.63, stdev=19107.88
>      lat (msec): min=11, max=412, avg=393.25, stdev=19.16
>     clat percentiles (msec):
>      |  1.00th=[  371],  5.00th=[  375], 10.00th=[  375], 20.00th=[  375],
>      | 30.00th=[  375], 40.00th=[  379], 50.00th=[  379], 60.00th=[  383],
>      | 70.00th=[  388], 80.00th=[  392], 90.00th=[  392], 95.00th=[  396],
>      | 99.00th=[  396], 99.50th=[  400], 99.90th=[  400], 99.95th=[  400],
>      | 99.99th=[  400]
>     lat (usec) : 10=0.02%
>     lat (msec) : 20=0.02%, 50=0.04%, 100=0.08%, 250=0.25%, 500=99.59%
>   cpu          : usr=2.22%, sys=30.01%, ctx=53455, majf=0, minf=25
>   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, >=64=0.0%
>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
>      issued    : total=r=0/w=4867/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
>      latency   : target=0, window=0, percentile=100.00%, depth=32
>
> Run status group 0 (all jobs):
>   WRITE: io=19468MB, aggrb=332248KB/s, minb=332248KB/s, maxb=332248KB/s, mint=60001msec, maxt=60001msec
>
> Disk stats (read/write):
>   sdd: ios=41/38830, merge=0/4931664, ticks=4/203944, in_queue=203700, util=79.55%
>   
>   

The disk seems to be properly detected properly by the OS:

> [4145961.899584] scsi 0:0:16:0: Direct-Access     ATA      SAMSUNG MZ7LH3T8 904Q PQ: 0 ANSI: 6
> [4145961.899596] scsi 0:0:16:0: SATA: handle(0x001a), sas_addr(0x500304801ef80883), phy(3), device_name(0x0000000000000000)
> [4145961.899598] scsi 0:0:16:0: enclosure logical id (0x500304801ef808bf), slot(3) 
> [4145961.899599] scsi 0:0:16:0: enclosure level(0x0000), connector name(     )
> [4145961.899649] scsi 0:0:16:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
> [4145961.900471] sd 0:0:16:0: Power-on or device reset occurred
> [4145961.900544] sd 0:0:16:0: Attached scsi generic sg3 type 0
> [4145961.902367] sd 0:0:16:0: [sdd] 7501476528 512-byte logical blocks: (3.84 TB/3.49 TiB)
> [4145961.902369] sd 0:0:16:0: [sdd] 4096-byte physical blocks
> [4145961.904264] sd 0:0:16:0: [sdd] Write Protect is off
> [4145961.904266] sd 0:0:16:0: [sdd] Mode Sense: 9b 00 10 08
> [4145961.904756] sd 0:0:16:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
> [4145961.915996] sd 0:0:16:0: [sdd] Attached SCSI disk

Anyone with a idea what we could be doing wrong? Or are these disks 
really unsuitable for OSD use?

MJ
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx