Hi, Maged
Not a big difference in both cases.
Performance of 4 nodes pool with 5x PM863a each is:
4k bs - 33-37kIOPS krbd 128 threads and 42-51kIOPS vs 1024
threads (fio numjobs 128-256-512)
the same situation happens when we try to increase rbd workload, 3 rbd gets the same iops #.
Dead end & limit )
Thank you!
2018-03-12 21:49 GMT+03:00 Maged Mokhtar <mmokhtar@xxxxxxxxxxx>:
Hi,
Try increasing the queue depth from default 128 to 1024:
rbd map image-XX -o queue_depth=1024
Also if you run multiple rbd images/fio tests, do you get higher combined performance ?
Maged
On 2018-03-12 17:16, Sergey Kotov wrote:
Dear moderator, i subscribed to ceph list today, could you please post my message?
---------- Forwarded message ----------
From: Sergey Kotov <graycep1@xxxxxxxxx>
Date: 2018-03-06 10:52 GMT+03:00
Subject: [ceph bad performance], can't find a bottleneck
To: ceph-users@xxxxxxxxxxxxxx
Cc: Житенев Алексей <zhitenevav@xxxxxxxxxx>, Anna Anikina <anikina@xxxxxxxxx>
Good day.
Can you please help us to find bottleneck in our ceph installations.We have 3 SSD-only clusters with different HW, but situation is the same - overall i/o operations between client & ceph lower than 1/6 of summary performance all ssd.
For example -
One of our cluster has 4-nodes with ssd Toshiba 2Tb Enterprise drives, installed on Ubuntu server 16.04.
Servers are connected to the 10G switches. Latency between modes is about 0.1ms. Ethernet utilisation is low.
# uname -a
Linux storage01 4.4.0-101-generic #124-Ubuntu SMP Fri Nov 10 18:29:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
# ceph osd versions
{
"ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable)": 55
}
When we map rbd image direct on the storage nodes via krbd, performance is not good enough.
We use fio for testing. Even we run randwrite with 4k block size test in multiple thread mode, our drives don't have utilisation higher then 30% and lat is ok.At the same time iostat tool displays 100% utilisation on /dev/rbdX.Sample fio tests:
Also we can't enable rbd_cache, because of using scst iscsi over rbd mapped images.
How can we resolve the issue?
Ceph config:
[global]
fsid = beX482fX-6a91-46dX-ad22-21a8a2696abX
mon_initial_members = storage01, storage02, storage03
mon_host = X.Y.Z.1,X.Y.Z.2,X.Y.Z.3
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = X.Y.Z.0/24
filestore_xattr_use_omap = true
osd_pool_default_size = 2
osd_pool_default_min_size = 1
osd_pool_default_pg_num = 1024
osd_journal_size = 10240
osd_mkfs_type = xfs
filestore_op_threads = 16
filestore_wbthrottle_enable = False
throttler_perf_counter = False
osd crush update on start = false
[osd]
osd_scrub_begin_hour = 1
osd_scrub_end_hour = 6
osd_scrub_priority = 1
osd_enable_op_tracker = False
osd_max_backfills = 1
osd heartbeat grace = 20
osd heartbeat interval = 5
osd recovery max active = 1
osd recovery max single start = 1
osd recovery op priority = 1
osd recovery threads = 1
osd backfill scan max = 16
osd backfill scan min = 4
osd max scrubs = 1
osd scrub interval randomize ratio = 1.0
osd disk thread ioprio class = idle
osd disk thread ioprio priority = 0
osd scrub chunk max = 1
osd scrub chunk min = 1
osd deep scrub stride = 1048576
osd scrub load threshold = 5.0
osd scrub sleep = 0.1
[client]
rbd_cache = false
root@storage04:~# fio --name iops --rw randread --bs 4k --filename /dev/rbd2 --numjobs 12 --ioengine=libaio --group_reportingiops: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1...fio-2.2.10Starting 12 processes^Cbs: 12 (f=12): [r(12)] [1.2% done] [128.4MB/0KB/0KB /s] [32.9K/0/0 iops] [eta 16m:49s]fio: terminating on signal 2iops: (groupid=0, jobs=12): err= 0: pid=29812: Sun Feb 11 23:59:19 2018read : io=1367.8MB, bw=126212KB/s, iops=31553, runt= 11097msecslat (usec): min=1, max=59700, avg=375.92, stdev=495.19clat (usec): min=0, max=377, avg= 1.12, stdev= 3.16lat (usec): min=1, max=59702, avg=377.61, stdev=495.32clat percentiles (usec):| 1.00th=[ 0], 5.00th=[ 0], 10.00th=[ 1], 20.00th=[ 1],| 30.00th=[ 1], 40.00th=[ 1], 50.00th=[ 1], 60.00th=[ 1],| 70.00th=[ 1], 80.00th=[ 1], 90.00th=[ 1], 95.00th=[ 2],| 99.00th=[ 2], 99.50th=[ 2], 99.90th=[ 73], 99.95th=[ 78],| 99.99th=[ 115]bw (KB /s): min= 8536, max=11944, per=8.33%, avg=10516.45, stdev=635.32lat (usec) : 2=91.74%, 4=7.93%, 10=0.14%, 20=0.09%, 50=0.01%lat (usec) : 100=0.07%, 250=0.03%, 500=0.01%cpu : usr=1.32%, sys=3.69%, ctx=329556, majf=0, minf=134IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%issued : total=r=350144/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0latency : target=0, window=0, percentile=100.00%, depth=1Run status group 0 (all jobs):READ: io=1367.8MB, aggrb=126212KB/s, minb=126212KB/s, maxb=126212KB/s, mint=11097msec, maxt=11097msecDisk stats (read/write):rbd2: ios=323072/0, merge=0/0, ticks=124268/0, in_queue=124680, util=99.31%root@storage04:~# fio --name iops --rw randwrite --bs 4k --filename /dev/rbd2 --numjobs 12 --ioengine=libaio --group_reportingiops: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1...fio-2.2.10Starting 12 processes^Cbs: 12 (f=12): [w(12)] [25.0% done] [0KB/713.5MB/0KB /s] [0/183K/0 iops] [eta 00m:45s]fio: terminating on signal 2fio: job 'iops' hasn't exited in 60 seconds, it appears to be stuck. Doing forceful exit of this job.iops: (groupid=0, jobs=12): err= 0: pid=30327: Mon Feb 12 00:01:04 2018write: io=32020MB, bw=2172.5MB/s, iops=556149, runt= 14739msecslat (usec): min=1, max=255011, avg=20.00, stdev=1110.12clat (usec): min=0, max=4426, avg= 0.35, stdev= 2.32lat (usec): min=1, max=255015, avg=20.43, stdev=1110.18clat percentiles (usec):| 1.00th=[ 0], 5.00th=[ 0], 10.00th=[ 0], 20.00th=[ 0],| 30.00th=[ 0], 40.00th=[ 0], 50.00th=[ 0], 60.00th=[ 0],| 70.00th=[ 1], 80.00th=[ 1], 90.00th=[ 1], 95.00th=[ 1],| 99.00th=[ 1], 99.50th=[ 1], 99.90th=[ 2], 99.95th=[ 3],| 99.99th=[ 81]bw (KB /s): min=16895, max=641688, per=8.68%, avg=193110.20, stdev=150217.09lat (usec) : 2=99.88%, 4=0.08%, 10=0.01%, 20=0.01%, 50=0.01%lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%lat (msec) : 10=0.01%cpu : usr=5.51%, sys=23.27%, ctx=8723, majf=0, minf=3528IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%issued : total=r=0/w=8197085/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0latency : target=0, window=0, percentile=100.00%, depth=1Run status group 0 (all jobs):WRITE: io=32020MB, aggrb=2172.5MB/s, minb=2172.5MB/s, maxb=2172.5MB/s, mint=14739msec, maxt=14739msecDisk stats (read/write):rbd2: ios=0/30044, merge=0/330950, ticks=0/1601972, in_queue=1603328, util=87.57%root@storage04:~# fio --name iops --rw randwrite --bs 4k --filename /dev/rbd2 --numjobs 12 --ioengine=libaio --group_reporting --direct=1iops: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1...fio-2.2.10Starting 12 processes^Cbs: 12 (f=12): [w(12)] [0.5% done] [0KB/14304KB/0KB /s] [0/3576/0 iops] [eta 03h:42m:23s]]fio: terminating on signal 2iops: (groupid=0, jobs=12): err= 0: pid=31552: Mon Feb 12 00:02:39 2018write: io=617132KB, bw=9554.8KB/s, iops=2388, runt= 64589msecslat (usec): min=3, max=463, avg=11.56, stdev=14.17clat (usec): min=930, max=999957, avg=5004.58, stdev=30299.43lat (usec): min=938, max=999975, avg=5016.50, stdev=30299.58clat percentiles (usec):| 1.00th=[ 1192], 5.00th=[ 1304], 10.00th=[ 1400], 20.00th=[ 1512],| 30.00th=[ 1608], 40.00th=[ 1720], 50.00th=[ 1832], 60.00th=[ 1960],| 70.00th=[ 2160], 80.00th=[ 2480], 90.00th=[ 4256], 95.00th=[12096],| 99.00th=[51968], 99.50th=[81408], 99.90th=[733184], 99.95th=[864256],| 99.99th=[946176]bw (KB /s): min= 4, max= 2251, per=9.30%, avg=888.88, stdev=453.38lat (usec) : 1000=0.02%lat (msec) : 2=62.38%, 4=26.86%, 10=5.16%, 20=1.82%, 50=2.66%lat (msec) : 100=0.75%, 250=0.22%, 500=0.03%, 750=0.01%, 1000=0.09%cpu : usr=0.24%, sys=0.54%, ctx=154351, majf=0, minf=149IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%issued : total=r=0/w=154283/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0latency : target=0, window=0, percentile=100.00%, depth=1Run status group 0 (all jobs):WRITE: io=617132KB, aggrb=9554KB/s, minb=9554KB/s, maxb=9554KB/s, mint=64589msec, maxt=64589msecDisk stats (read/write):rbd2: ios=0/154277, merge=0/0, ticks=0/767860, in_queue=768740, util=99.94%root@storage04:~# fio --name iops --rw randwrite --bs 4k --filename /dev/rbd2 --numjobs 12 --iodepth=32 --ioengine=libaio --group_reporting --direct=1 --runtime=100iops: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32...fio-2.2.10Starting 12 processesJobs: 12 (f=12): [w(12)] [100.0% done] [0KB/49557KB/0KB /s] [0/12.4K/0 iops] [eta 00m:00s]iops: (groupid=0, jobs=12): err= 0: pid=32632: Mon Feb 12 00:04:51 2018write: io=3093.6MB, bw=31675KB/s, iops=7918, runt=100009msecslat (usec): min=2, max=973942, avg=1510.77, stdev=18783.09clat (msec): min=1, max=1719, avg=46.97, stdev=111.64lat (msec): min=1, max=1719, avg=48.49, stdev=113.62clat percentiles (msec):| 1.00th=[ 5], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 13],| 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 21], 60.00th=[ 25],| 70.00th=[ 31], 80.00th=[ 43], 90.00th=[ 68], 95.00th=[ 131],| 99.00th=[ 742], 99.50th=[ 824], 99.90th=[ 963], 99.95th=[ 988],| 99.99th=[ 1045]bw (KB /s): min= 6, max= 8320, per=9.51%, avg=3011.84, stdev=1956.64lat (msec) : 2=0.07%, 4=0.55%, 10=10.28%, 20=37.47%, 50=35.89%lat (msec) : 100=9.47%, 250=3.45%, 500=0.61%, 750=1.26%, 1000=0.92%lat (msec) : 2000=0.03%cpu : usr=0.28%, sys=0.58%, ctx=265641, majf=0, minf=146IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%issued : total=r=0/w=791939/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0latency : target=0, window=0, percentile=100.00%, depth=32Run status group 0 (all jobs):WRITE: io=3093.6MB, aggrb=31674KB/s, minb=31674KB/s, maxb=31674KB/s, mint=100009msec, maxt=100009msecDisk stats (read/write):rbd2: ios=0/789529, merge=0/0, ticks=0/12345428, in_queue=12383232, util=100.00%root@storage04:~# fio --name iops --rw randwrite --bs 21k --filename /dev/rbd2 --numjobs 64 --iodepth=4 --ioengine=libaio --group_reporting --direct=1 --runtime=100iops: (g=0): rw=randwrite, bs=21K-21K/21K-21K/21K-21K, ioengine=libaio, iodepth=4...fio-2.2.10Starting 64 processes^Cbs: 64 (f=64): [w(64)] [28.7% done] [0KB/98028KB/0KB /s] [0/4668/0 iops] [eta 01m:12s]fio: terminating on signal 2Jobs: 9 (f=9): [E(1),w(1),_(2),w(1),_(3),w(2),_(2),w(1),_(15),w(1),_(7),w(1 ),_(13),w(1),_(2),w(1),_(10)] [1.9% done] [0KB/4804KB/0KB /s] [0/228/0 iops] [eta 27m:16s] iops: (groupid=0, jobs=64): err= 0: pid=5221: Mon Feb 12 00:10:52 2018write: io=1967.9MB, bw=67393KB/s, iops=3209, runt= 29900msecslat (usec): min=3, max=1211.3K, avg=14387.91, stdev=58236.34clat (msec): min=1, max=1711, avg=63.67, stdev=141.68lat (msec): min=1, max=1711, avg=78.06, stdev=159.64clat percentiles (usec):| 1.00th=[ 1944], 5.00th=[ 2864], 10.00th=[ 4192], 20.00th=[ 7136],| 30.00th=[11200], 40.00th=[16768], 50.00th=[23680], 60.00th=[33536],| 70.00th=[46848], 80.00th=[66048], 90.00th=[110080], 95.00th=[205824],| 99.00th=[790528], 99.50th=[872448], 99.90th=[1253376], 99.95th=[1368064],| 99.99th=[1531904]bw (KB /s): min= 23, max= 3906, per=1.77%, avg=1189.80, stdev=837.25lat (msec) : 2=1.25%, 4=8.01%, 10=18.08%, 20=17.69%, 50=27.03%lat (msec) : 100=16.47%, 250=6.99%, 500=0.94%, 750=2.15%, 1000=1.20%lat (msec) : 2000=0.19%cpu : usr=0.06%, sys=0.11%, ctx=173237, majf=0, minf=787IO depths : 1=0.1%, 2=0.1%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%issued : total=r=0/w=95955/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0latency : target=0, window=0, percentile=100.00%, depth=4Run status group 0 (all jobs):WRITE: io=1967.9MB, aggrb=67393KB/s, minb=67393KB/s, maxb=67393KB/s, mint=29900msec, maxt=29900msecDisk stats (read/write):rbd2: ios=0/95945, merge=0/0, ticks=0/3672960, in_queue=3685732, util=99.66%root@storage04:~# fio --name iops --rw randwrite --bs 21k --filename /dev/rbd2 --numjobs 32 --iodepth=4 --ioengine=libaio --group_reporting --direct=1 --runtime=100iops: (g=0): rw=randwrite, bs=21K-21K/21K-21K/21K-21K, ioengine=libaio, iodepth=4...fio-2.2.10Starting 32 processes^Cbs: 32 (f=32): [w(32)] [11.9% done] [0KB/40866KB/0KB /s] [0/1946/0 iops] [eta 01m:29s]fio: terminating on signal 2Jobs: 32 (f=30): [w(32)] [12.9% done] [0KB/53676KB/0KB /s] [0/2556/0 iops] [eta 01m:28s]iops: (groupid=0, jobs=32): err= 0: pid=6344: Mon Feb 12 00:12:01 2018write: io=820680KB, bw=66912KB/s, iops=3186, runt= 12265msecslat (usec): min=3, max=842, avg=13.14, stdev=15.18clat (msec): min=1, max=1270, avg=38.67, stdev=114.69lat (msec): min=1, max=1270, avg=38.68, stdev=114.69clat percentiles (usec):| 1.00th=[ 1976], 5.00th=[ 2416], 10.00th=[ 3024], 20.00th=[ 4384],| 30.00th=[ 6048], 40.00th=[ 8032], 50.00th=[10688], 60.00th=[14400],| 70.00th=[19584], 80.00th=[30080], 90.00th=[57088], 95.00th=[112128],| 99.00th=[765952], 99.50th=[839680], 99.90th=[954368], 99.95th=[987136],| 99.99th=[1089536]bw (KB /s): min= 30, max= 6300, per=3.58%, avg=2397.65, stdev=1548.34lat (msec) : 2=1.16%, 4=16.01%, 10=30.33%, 20=22.95%, 50=17.40%lat (msec) : 100=6.82%, 250=1.96%, 500=1.46%, 750=0.82%, 1000=1.07%lat (msec) : 2000=0.03%cpu : usr=0.13%, sys=0.21%, ctx=38649, majf=0, minf=372IO depths : 1=0.1%, 2=0.2%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%issued : total=r=0/w=39080/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0latency : target=0, window=0, percentile=100.00%, depth=4Run status group 0 (all jobs):WRITE: io=820680KB, aggrb=66912KB/s, minb=66912KB/s, maxb=66912KB/s, mint=12265msec, maxt=12265msecDisk stats (read/write):rbd2: ios=0/39077, merge=0/0, ticks=0/1504048, in_queue=1507272, util=99.17%root@storage04:~# fio --name iops --rw randwrite --bs 21k --filename /dev/rbd2 --numjobs 8 --iodepth=12 --ioengine=libaio --group_reporting --direct=1 --runtime=100iops: (g=0): rw=randwrite, bs=21K-21K/21K-21K/21K-21K, ioengine=libaio, iodepth=12...fio-2.2.10Starting 8 processes^Cbs: 8 (f=8): [w(8)] [20.8% done] [0KB/34650KB/0KB /s] [0/1650/0 iops] [eta 01m:20s]fio: terminating on signal 2Jobs: 2 (f=2): [_(5),w(2),_(1)] [10.1% done] [0KB/1554KB/0KB /s] [0/74/0 iops] [eta 03m:24s]iops: (groupid=0, jobs=8): err= 0: pid=6770: Mon Feb 12 00:12:32 2018write: io=979755KB, bw=44563KB/s, iops=2122, runt= 21986msecslat (usec): min=3, max=551, avg=12.92, stdev=14.78clat (msec): min=1, max=1531, avg=44.14, stdev=136.44lat (msec): min=1, max=1531, avg=44.15, stdev=136.44clat percentiles (usec):| 1.00th=[ 1944], 5.00th=[ 2320], 10.00th=[ 2672], 20.00th=[ 3568],| 30.00th=[ 4768], 40.00th=[ 6240], 50.00th=[ 8640], 60.00th=[12224],| 70.00th=[18816], 80.00th=[31872], 90.00th=[67072], 95.00th=[154624],| 99.00th=[839680], 99.50th=[913408], 99.90th=[1073152], 99.95th=[1368064],| 99.99th=[1466368]bw (KB /s): min= 26, max=18023, per=14.47%, avg=6449.72, stdev=5032.09lat (msec) : 2=1.37%, 4=22.49%, 10=30.69%, 20=16.94%, 50=15.12%lat (msec) : 100=6.72%, 250=3.13%, 500=0.73%, 750=1.14%, 1000=1.41%lat (msec) : 2000=0.25%cpu : usr=0.32%, sys=0.61%, ctx=44124, majf=0, minf=97IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0%submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%issued : total=r=0/w=46655/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0latency : target=0, window=0, percentile=100.00%, depth=12Run status group 0 (all jobs):WRITE: io=979755KB, aggrb=44562KB/s, minb=44562KB/s, maxb=44562KB/s, mint=21986msec, maxt=21986msecDisk stats (read/write):rbd2: ios=0/46652, merge=0/0, ticks=0/2037428, in_queue=2043236, util=99.65%
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
