Re: more performance issues :(

[Somnath Roy <Somnath.Roy@xxxxxxxxxxx>]

FYI, osd_op_threads is not used in the main io path anymore (from Giant). I don’t think increasing this will do any good.

If you want to tweak threads in io path play with the following two.

 

osd_op_num_threads_per_shard

osd_op_num_shards

 

But, It may not be the problem with writes..Default value should work just fine..Need some more info..

 

1. Are you using fio-rbd ? If so, try running with rbd_cache = false in the client side ceph.conf and see if that is making any difference.

 

2. What is the block size you are trying with ?

 

3. Check how the SSD is behaving with raw fio o_direct and o_dsync mode with the same block size

 

4. What kind of fio write io profile are you running ? Hope you are doing similar IO profile with benchwrite.

 

5. How many OSDs a single SSD as journal is serving ? How many OSDs total you are running ? what is the replication factor ?

 

6. Hope none of the resources are saturating

 

Thanks & Regards

Somnath

 

 

From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Tyler Bishop
Sent: Saturday, December 26, 2015 8:38 AM
To: Florian Rommel
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re: [ceph-users] more performance issues :(

 

Add this under osd.

 

osd op threads = 8

Restart the osd services and try that.

 

  

 

---

From: "Florian Rommel" <florian.rommel@xxxxxxxxxxxxxxx>
To: "Wade Holler" <wade.holler@xxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx
Sent: Saturday, December 26, 2015 4:55:06 AM
Subject: Re: [ceph-users] more performance issues :(

 

Hi, iostat shows all OSDs working when data is benched.  it looks like the culprit is nowhere to be found. If i add SSD journals with the SSDs that we have, even thought they give a much higher result with fio than the SATA drives, the speed of the cluster is exactly the same… 150-180MB/s, while reads max out the 10GBe network with no problem.

rbd benchwrite however gives me NICE throughput… about 500MB /s to start with and then dropping and flattening out at 320MB/s, 90000 IOPs…. so what the hell is going on?.

 

 

if i take the journals off and move them to the disks themselves, same results. Something is really really off with my config i guess, and I need to do some serious troubleshooting to figure this out.

 

Thanks for the help so far .

//Florian

 

 

 

On 24 Dec 2015, at 13:54, Wade Holler <wade.holler@xxxxxxxxx> wrote:

 

Have a look at the iostsat -x 1 1000 output to see what the drives are doing

 

On Wed, Dec 23, 2015 at 4:35 PM Florian Rommel <florian.rommel@xxxxxxxxxxxxxxx> wrote:

Ah, totally forgot the additional details :)

OS is SUSE Enterprise Linux 12.0 with all patches,
Ceph version 0.94.3
4 node cluster with 2x 10GBe networking, one for cluster and one for public network, 1 additional server purely as an admin server.
Test machine is also 10gbe connected

ceph.conf is included:
[global]
fsid = 312e0996-a13c-46d3-abe3-903e0b4a589a
mon_initial_members = ceph-admin, ceph-01, ceph-02, ceph-03, ceph-04
mon_host = 192.168.0.190,192.168.0.191,192.168.0.192,192.168.0.193,192.168.0.194
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
public network = 192.168.0.0/24
cluster network = 192.168.10.0/24

osd pool default size = 2
[osd]
osd journal size = 2048

Thanks again for any help and merry xmas already .
//F
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com