HI,
Ceph.conf:
osd journal size = 15360
rbd cache = true
rbd cache size = 2147483648
rbd cache max dirty = 1073741824
rbd cache max dirty age = 100
osd recovery max active = 1
osd max backfills = 1
osd mkfs options xfs = "-f -i size=2048"
osd mount options xfs = "rw,noatime,nobarrier,logbsize=256k,logbufs=8,inode64,allocsize=4M"
osd op threads = 8
so it should be 8 threads?
All 3 machines have more or less the same disk load at the same time.
also the disks:
sdb 35.56 87.10 6849.09 617310 48540806
sdc 26.75 72.62 5148.58 514701 36488992
sdd 35.15 53.48 6802.57 378993 48211141
sde 31.04 79.04 6208.48 560141 44000710
sdf 32.79 38.35 6238.28 271805 44211891
sdg 31.67 77.84 5987.45 551680 42434167
sdh 32.95 51.29 6315.76 363533 44761001
sdi 31.67 56.93 5956.29 403478 42213336
sdj 35.83 77.82 6929.31 551501 49109354
sdk 36.86 73.84 7291.00 523345 51672704
sdl 36.02 112.90 7040.47 800177 49897132
sdm 33.25 38.02 6455.05 269446 45748178
sdn 33.52 39.10 6645.19 277101 47095696
sdo 33.26 46.22 6388.20 327541 45274394
sdp 33.38 74.12 6480.62 525325 45929369
the question is: is this a poor performance to get max 500mb/write with 45 disks and replica 2 or should I expect this?
-----Urspr?ngliche Nachricht-----
Von: ceph-users [mailto:ceph-users-bounces at lists.ceph.com] Im Auftrag von Wido den Hollander
Gesendet: Donnerstag, 03. Juli 2014 15:22
An: ceph-users at lists.ceph.com
Betreff: Re: [ceph-users] write performance per disk
On 07/03/2014 03:11 PM, VELARTIS Philipp D?rhammer wrote:
> Hi,
>
> I have a ceph cluster setup (with 45 sata disk journal on disks) and
> get only 450mb/sec writes seq (maximum playing around with threads in
> rados
> bench) with replica of 2
>
How many threads?
> Which is about ~20Mb writes per disk (what y see in atop also)
> theoretically with replica2 and having journals on disk should be 45 X
> 100mb (sata) / 2 (replica) / 2 (journal writes) which makes it 1125
> satas in reality have 120mb/sec so the theoretical output should be more.
>
> I would expect to have between 40-50mb/sec for each sata disk
>
> Can somebody confirm that he can reach this speed with a setup with
> journals on the satas (with journals on ssd speed should be 100mb per disk)?
> or does ceph only give about ? of the speed for a disk? (and not the ?
> as expected because of journals)
>
Did you verify how much each machine is doing? It could be that the data is not distributed evenly and that on a certain machine the drives are doing 50MB/sec.
> My setup is 3 servers with: 2 x 2.6ghz xeons, 128gb ram 15 satas for
> ceph (and ssds for system) 1 x 10gig for external traffic, 1 x 10gig
> for osd traffic with reads I can saturate the network but writes is
> far away. And I would expect at least to saturate the 10gig with
> sequential writes also
>
Should be possible, but with 3 servers the data distribution might not be optimal causing a lower write performance.
I've seen 10Gbit write performance on multiple clusters without any problems.
> Thank you
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Wido den Hollander
Ceph consultant and trainer
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
_______________________________________________
ceph-users mailing list
ceph-users at lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
