-----Original Message-----
From: mj <lists@xxxxxxxxxxxxx>
Sent: 19 January 2021 14:33
To: ceph-users <ceph-users@xxxxxxx>
Subject: Samsung PM883 3.84TB SSD performance
Hi,
We purchased Samsung PM883 3.84TB SSDs to be used as BlueStore SSDs in
our cluster.
I ran some benchmarks (write cache disabled, and with -sync=1 and
-fsync=1) but with my little understandig, the results look terribly
bad. (iops around 82!)
Reading pages like yourcmc.ru, we should at least aim for > 1000s IOPS.
(like 10000 up to 20000)
Here is our complete output:
root@pve:~# hdparm -W 0 /dev/sdd
/dev/sdd:
setting drive write-caching to 0 (off)
write-caching = 0 (off)
root@pve:~# fio -ioengine=libaio -fsync=1 -invalidate=1 -name=test -
bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd
test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio,
iodepth=32
fio-2.16
Starting 1 process
Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta
00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=3268778: Tue Jan 19 14:13:44
2021
write: io=19728MB, bw=336674KB/s, iops=82, runt= 60003msec
slat (usec): min=1853, max=4652, avg=2428.94, stdev=361.92
clat (msec): min=9, max=652, avg=376.87, stdev=22.88
lat (msec): min=11, max=655, avg=379.30, stdev=22.94
clat percentiles (msec):
| 1.00th=[ 363], 5.00th=[ 367], 10.00th=[ 371], 20.00th=[
371],
| 30.00th=[ 371], 40.00th=[ 375], 50.00th=[ 375], 60.00th=[
379],
| 70.00th=[ 379], 80.00th=[ 383], 90.00th=[ 388], 95.00th=[
392],
| 99.00th=[ 404], 99.50th=[ 420], 99.90th=[ 611], 99.95th=[
635],
| 99.99th=[ 652]
lat (msec) : 10=0.02%, 50=0.06%, 100=0.10%, 250=0.24%, 500=99.25%
lat (msec) : 750=0.32%
cpu : usr=2.95%, sys=28.30%, ctx=47835, majf=0, minf=26
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%,
=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
=64=0.0%
issued : total=r=0/w=4932/d=0, short=r=0/w=0/d=0,
drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: io=19728MB, aggrb=336674KB/s, minb=336674KB/s,
maxb=336674KB/s, mint=60003msec, maxt=60003msec
Disk stats (read/write):
sdd: ios=41/39330, merge=0/4995672, ticks=8/231816, in_queue=231440,
util=79.43%
root@pve:~# fio -ioengine=libaio -sync=1 -invalidate=1 -name=test -
bs=4M -iodepth=32 -rw=randwrite -runtime=60 -filename=/dev/sdd
test: (g=0): rw=randwrite, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio,
iodepth=32
fio-2.16
Starting 1 process
Jobs: 1 (f=1): [f(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta
00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=3270282: Tue Jan 19 14:14:58
2021
write: io=19468MB, bw=332248KB/s, iops=81, runt= 60001msec
slat (usec): min=11658, max=14287, avg=12318.59, stdev=371.39
clat (usec): min=8, max=400078, avg=380931.63, stdev=19107.88
lat (msec): min=11, max=412, avg=393.25, stdev=19.16
clat percentiles (msec):
| 1.00th=[ 371], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[
375],
| 30.00th=[ 375], 40.00th=[ 379], 50.00th=[ 379], 60.00th=[
383],
| 70.00th=[ 388], 80.00th=[ 392], 90.00th=[ 392], 95.00th=[
396],
| 99.00th=[ 396], 99.50th=[ 400], 99.90th=[ 400], 99.95th=[
400],
| 99.99th=[ 400]
lat (usec) : 10=0.02%
lat (msec) : 20=0.02%, 50=0.04%, 100=0.08%, 250=0.25%, 500=99.59%
cpu : usr=2.22%, sys=30.01%, ctx=53455, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=99.4%,
=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%,
=64=0.0%
issued : total=r=0/w=4867/d=0, short=r=0/w=0/d=0,
drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
WRITE: io=19468MB, aggrb=332248KB/s, minb=332248KB/s,
maxb=332248KB/s, mint=60001msec, maxt=60001msec
Disk stats (read/write):
sdd: ios=41/38830, merge=0/4931664, ticks=4/203944, in_queue=203700,
util=79.55%
The disk seems to be properly detected properly by the OS:
[4145961.899584] scsi 0:0:16:0: Direct-Access ATA SAMSUNG
MZ7LH3T8 904Q PQ: 0 ANSI: 6
[4145961.899596] scsi 0:0:16:0: SATA: handle(0x001a),
sas_addr(0x500304801ef80883), phy(3), device_name(0x0000000000000000)
[4145961.899598] scsi 0:0:16:0: enclosure logical id
(0x500304801ef808bf), slot(3)
[4145961.899599] scsi 0:0:16:0: enclosure level(0x0000), connector
name( )
[4145961.899649] scsi 0:0:16:0: atapi(n), ncq(y), asyn_notify(n),
smart(y), fua(y), sw_preserve(y)
[4145961.900471] sd 0:0:16:0: Power-on or device reset occurred
[4145961.900544] sd 0:0:16:0: Attached scsi generic sg3 type 0
[4145961.902367] sd 0:0:16:0: [sdd] 7501476528 512-byte logical
blocks: (3.84 TB/3.49 TiB)
[4145961.902369] sd 0:0:16:0: [sdd] 4096-byte physical blocks
[4145961.904264] sd 0:0:16:0: [sdd] Write Protect is off
[4145961.904266] sd 0:0:16:0: [sdd] Mode Sense: 9b 00 10 08
[4145961.904756] sd 0:0:16:0: [sdd] Write cache: enabled, read cache:
enabled, supports DPO and FUA
[4145961.915996] sd 0:0:16:0: [sdd] Attached SCSI disk
Anyone with a idea what we could be doing wrong? Or are these disks
really unsuitable for OSD use?
MJ
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
