Re: Samsung PM883 3.84TB SSD performance

[Igor Fedotov <ifedotov@xxxxxxx>]

I suspect you meant >1000s IOPS for 4K blocks, not 4M ones.

Hence -bs=4K not -bs=4M.

On 1/19/2021 4:32 PM, mj wrote:
> Hi,
>
> We purchased Samsung PM883 3.84TB SSDs to be used as BlueStore SSDs in 
> our cluster.
>
> I ran some benchmarks (write cache disabled, and with -sync=1 and 
> -fsync=1) but with my little understandig, the results look terribly 
> bad. (iops around 82!)
>
> Reading pages like yourcmc.ru, we should at least aim for > 1000s 
> IOPS. (like 10000 up to 20000)
>
> Here is our complete output:
>
> > root@pve:~# hdparm -W 0 /dev/sdd
> >
> > /dev/sdd:
> >  setting drive write-caching to 0 (off)
> >  write-caching =  0 (off)
> > root@pve:~# fio -ioengine=libaio -fsync=1 -invalidate=1 -name=test 
> > -bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd
> > test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, 
> > iodepth=32
> > fio-2.16
> > Starting 1 process
> > Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] 
> > [eta 00m:00s]     test: (groupid=0, jobs=1): err= 0: pid=3268778: Tue 
> > Jan 19 14:13:44 2021
> >   write: io=19728MB, bw=336674KB/s, iops=82, runt= 60003msec
> >     slat (usec): min=1853, max=4652, avg=2428.94, stdev=361.92
> >     clat (msec): min=9, max=652, avg=376.87, stdev=22.88
> >      lat (msec): min=11, max=655, avg=379.30, stdev=22.94
> >     clat percentiles (msec):
> >      |  1.00th=[  363],  5.00th=[  367], 10.00th=[  371], 20.00th=[  
> > 371],
> >      | 30.00th=[  371], 40.00th=[  375], 50.00th=[  375], 60.00th=[  
> > 379],
> >      | 70.00th=[  379], 80.00th=[  383], 90.00th=[  388], 95.00th=[  
> > 392],
> >      | 99.00th=[  404], 99.50th=[  420], 99.90th=[  611], 99.95th=[  
> > 635],
> >      | 99.99th=[  652]
> >     lat (msec) : 10=0.02%, 50=0.06%, 100=0.10%, 250=0.24%, 500=99.25%
> >     lat (msec) : 750=0.32%
> >   cpu          : usr=2.95%, sys=28.30%, ctx=47835, majf=0, minf=26
> >   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, 
> > >=64=0.0%
> >      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
> > >=64=0.0%
> >      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, 
> > >=64=0.0%
> >      issued    : total=r=0/w=4932/d=0, short=r=0/w=0/d=0, 
> > drop=r=0/w=0/d=0
> >      latency   : target=0, window=0, percentile=100.00%, depth=32
> >
> > Run status group 0 (all jobs):
> >   WRITE: io=19728MB, aggrb=336674KB/s, minb=336674KB/s, 
> > maxb=336674KB/s, mint=60003msec, maxt=60003msec
> >
> > Disk stats (read/write):
> >   sdd: ios=41/39330, merge=0/4995672, ticks=8/231816, 
> > in_queue=231440, util=79.43%
> > root@pve:~# fio -ioengine=libaio -sync=1 -invalidate=1 -name=test 
> > -bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd
> > test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, 
> > iodepth=32
> > fio-2.16
> > Starting 1 process
> > Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] 
> > [eta 00m:00s]     test: (groupid=0, jobs=1): err= 0: pid=3270282: Tue 
> > Jan 19 14:14:58 2021
> >   write: io=19468MB, bw=332248KB/s, iops=81, runt= 60001msec
> >     slat (usec): min=11658, max=14287, avg=12318.59, stdev=371.39
> >     clat (usec): min=8, max=400078, avg=380931.63, stdev=19107.88
> >      lat (msec): min=11, max=412, avg=393.25, stdev=19.16
> >     clat percentiles (msec):
> >      |  1.00th=[  371],  5.00th=[  375], 10.00th=[  375], 20.00th=[  
> > 375],
> >      | 30.00th=[  375], 40.00th=[  379], 50.00th=[  379], 60.00th=[  
> > 383],
> >      | 70.00th=[  388], 80.00th=[  392], 90.00th=[  392], 95.00th=[  
> > 396],
> >      | 99.00th=[  396], 99.50th=[  400], 99.90th=[  400], 99.95th=[  
> > 400],
> >      | 99.99th=[  400]
> >     lat (usec) : 10=0.02%
> >     lat (msec) : 20=0.02%, 50=0.04%, 100=0.08%, 250=0.25%, 500=99.59%
> >   cpu          : usr=2.22%, sys=30.01%, ctx=53455, majf=0, minf=25
> >   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%, 
> > >=64=0.0%
> >      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
> > >=64=0.0%
> >      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, 
> > >=64=0.0%
> >      issued    : total=r=0/w=4867/d=0, short=r=0/w=0/d=0, 
> > drop=r=0/w=0/d=0
> >      latency   : target=0, window=0, percentile=100.00%, depth=32
> >
> > Run status group 0 (all jobs):
> >   WRITE: io=19468MB, aggrb=332248KB/s, minb=332248KB/s, 
> > maxb=332248KB/s, mint=60001msec, maxt=60001msec
> >
> > Disk stats (read/write):
> >   sdd: ios=41/38830, merge=0/4931664, ticks=4/203944, 
> > in_queue=203700, util=79.55%
>
> The disk seems to be properly detected properly by the OS:
>
> > [4145961.899584] scsi 0:0:16:0: Direct-Access     ATA      SAMSUNG 
> > MZ7LH3T8 904Q PQ: 0 ANSI: 6
> > [4145961.899596] scsi 0:0:16:0: SATA: handle(0x001a), 
> > sas_addr(0x500304801ef80883), phy(3), device_name(0x0000000000000000)
> > [4145961.899598] scsi 0:0:16:0: enclosure logical id 
> > (0x500304801ef808bf), slot(3) [4145961.899599] scsi 0:0:16:0: 
> > enclosure level(0x0000), connector name(     )
> > [4145961.899649] scsi 0:0:16:0: atapi(n), ncq(y), asyn_notify(n), 
> > smart(y), fua(y), sw_preserve(y)
> > [4145961.900471] sd 0:0:16:0: Power-on or device reset occurred
> > [4145961.900544] sd 0:0:16:0: Attached scsi generic sg3 type 0
> > [4145961.902367] sd 0:0:16:0: [sdd] 7501476528 512-byte logical 
> > blocks: (3.84 TB/3.49 TiB)
> > [4145961.902369] sd 0:0:16:0: [sdd] 4096-byte physical blocks
> > [4145961.904264] sd 0:0:16:0: [sdd] Write Protect is off
> > [4145961.904266] sd 0:0:16:0: [sdd] Mode Sense: 9b 00 10 08
> > [4145961.904756] sd 0:0:16:0: [sdd] Write cache: enabled, read cache: 
> > enabled, supports DPO and FUA
> > [4145961.915996] sd 0:0:16:0: [sdd] Attached SCSI disk
>
> Anyone with a idea what we could be doing wrong? Or are these disks 
> really unsuitable for OSD use?
>
> MJ
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx