You have to tell a bit about your cluster setup, like nr of osd's, 3x replication on your testing pool?
Eg. this[1] was my test on a cluster with only 1gbit ethernet, 3x repl hdd pool. This[2] with 10gbit and more osd's added
[2]
[root@c01 ~]# rados bench -p rbd 10 write
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_c01_3576497
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 41 25 99.9948 100 0.198773 0.41148
2 16 101 85 169.984 240 0.203578 0.347027
3 16 172 156 207.979 284 0.0863202 0.296866
4 16 245 229 228.975 292 0.139681 0.268933
5 16 322 306 244.772 308 0.107296 0.257353
6 16 385 369 245.97 252 0.601879 0.250782
7 16 460 444 253.684 300 0.154803 0.247178
8 16 541 525 262.467 324 0.274302 0.241951
9 16 604 588 261.3 252 0.11929 0.238717
10 16 672 656 262.367 272 0.134654 0.241424
Total time run: 10.1504
Total writes made: 673
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 265.212
Stddev Bandwidth: 63.0823
Max bandwidth (MB/sec): 324
Min bandwidth (MB/sec): 100
Average IOPS: 66
Stddev IOPS: 15.7706
Max IOPS: 81
Min IOPS: 25
Average Latency(s): 0.241012
Stddev Latency(s): 0.154282
Max latency(s): 1.05851
Min latency(s): 0.0702826
Cleaning up (deleting benchmark objects)
Removed 673 objects
Clean up completed and total clean up time :1.26346
[1]
[@]# rados bench -p rbd 10 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects
Object prefix: benchmark_data_c01_18283
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 27 11 43.9884 44 0.554119 0.624979
2 16 47 31 61.9841 80 1.04112 0.793553
3 16 57 41 54.654 40 1.33104 0.876273
4 16 75 59 58.9869 72 0.840098 0.97091
5 16 97 81 64.7864 88 1.02915 0.922043
6 16 105 89 59.3207 32 1.2471 0.915408
7 16 129 113 64.5582 96 0.616579 0.947882
8 16 145 129 64.4866 64 1.09397 0.921441
9 16 163 147 65.3201 72 0.885566 0.906388
10 16 166 150 59.9881 12 1.22834 0.909591
11 13 167 154 55.9889 16 2.30029 0.942798
Total time run: 11.141939
Total writes made: 167
Write size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 59.9537
Stddev Bandwidth: 28.7889
Max bandwidth (MB/sec): 96
Min bandwidth (MB/sec): 12
Average IOPS: 14
Stddev IOPS: 7
Max IOPS: 24
Min IOPS: 3
Average Latency(s): 1.06157
Stddev Latency(s): 0.615773
Max latency(s): 3.23088
Min latency(s): 0.171585
> -----Original Message-----
> Sent: 10 February 2021 10:14
> To: Marc <Marc@xxxxxxxxxxxxxxxxx>
> Cc: ceph-users <ceph-users@xxxxxxx>
> Subject: Re: struggling to achieve high bandwidth on Ceph
> dev cluster - HELP
>
> thanks for the reply.
>
> Yes, 4MB is the default. I have tried it. For example below (posted) is
> for
> 4MB (default) ran for 600 seconds. The seq read and rand read gives me a
> good bandwidth (not posted here). But with write its still very less.
> And I
> am particularly interested in block sizes. And rados bench tool has
> block
> size option which I have been using.
>
> Total time run: 601.106
> Total writes made: 2966
> Write size: 4194304
> Object size: 4194304
> Bandwidth (MB/sec): 19.7369
> Stddev Bandwidth: 14.8408
> Max bandwidth (MB/sec): 64
> Min bandwidth (MB/sec): 0
> Average IOPS: 4
> Stddev IOPS: 3.67408
> Max IOPS: 16
> Min IOPS: 0
> Average Latency(s): 3.24064
> Stddev Latency(s): 2.75111
> Max latency(s): 42.4551
> Min latency(s): 0.167701
>
> On Wed, Feb 10, 2021 at 9:46 AM Marc <Marc@xxxxxxxxxxxxxxxxx> wrote:
>
> >
> > try 4MB that is the default not?
> >
> >
> >
> > > -----Original Message-----
> > > Sent: 10 February 2021 09:30
> > > To: ceph-users <ceph-users@xxxxxxx>; dev <dev@xxxxxxx>; ceph-
> qa@xxxxxxx
> > > Subject: struggling to achieve high bandwidth on Ceph
> dev
> > > cluster - HELP
> > >
> > > Hi,
> > >
> > > Hello I am using rados bench tool. Currently I am using this tool
> on
> > > the
> > > development cluster after running vstart.sh script. It is working
> fine
> > > and
> > > I am interested in benchmarking the cluster. However I am struggling
> to
> > > achieve a good bandwidth i.e. bandwidth (MB/sec). My target
> throughput
> > > is
> > > at least 50 MB/sec and more. But mostly I am achieving is around 15-
> 20
> > > MB/sec. So, very poor.
> > >
> > > I am quite sure I am missing something. Either I have to change my
> > > cluster
> > > through vstart.sh script or I am not fully utilizing the rados bench
> > > tool.
> > > Or may be both. i.e. not the right cluster and also not using the
> rados
> > > bench tool correctly.
> > >
> > > Some of the shell examples I have been using to build the cluster
> are
> > > bellow:
> > > MDS=0 RGW=1 ../src/vstart.sh -d -l -n --bluestore
> > > MDS=0 RGW=1 MON=1 OSD=4../src/vstart.sh -d -l -n --bluestore
> > >
> > > While using rados bench tool I have been trying with different block
> > > sizes
> > > 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K. And I have also been
> changing
> > > the
> > > -t parameter in the shell to increase concurrent IOs.
> > >
> >
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
