Performance issues on small cluster

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Guys,

 

I’m in the process of configuring a ceph cluster and am getting some less than ideal performance and need some help figuring it out!

 

This cluster will only really be used for backup storage for Veeam so I don’t need a crazy amount of I/O but good sequential writes would be ideal.

 

At the moment I am only getting about 50MBs according to rados bench.

 

Our environment looks like this.

 

3x OSD Servers:

·         1x E3 CPU

·         24 GB RAM

·         8x 3TB Enterprise Drives

·         2x SSDs for journals – I haven’t used enterprise ones, just whatever I had around the office. I figure that this could be the bottle neck but I’d figure I’d get more than 50MBs.

·         1x 2 port 10GB NIC with separate public / replication networks

 

3x Monitor Servers

·         2x Quad

·         16GB RAM

·         2x 300GB SAS RAID 1

·         1x 2 port 10GB NIC

 

My results from bench tests are:

 

DD on the RBD device is showing ~50-60MBs

DD on the disks is showing ~160MBs

Rados Bench is showing ~50MBs

Network is showing 9.6Gbits on all servers

FIO is showing the below:

 

  WRITE: io=1024.0MB, aggrb=3099KB/s, minb=3099KB/s, maxb=3099KB/s, mint=338300msec, maxt=338300msec

 

 

My ceph config is as follows:

 

mon_host = 192.168.78.101,192.168.78.102,192.168.78.103

public_network = 192.168.78.0/24

cluster_network = 192.168.79.0/24

ms_bind_ipv6 = false

max_open_files = 131072

auth_cluster_required = cephx

auth_service_required = cephx

auth_client_required = cephx

osd_journal_size = 10000

osd_pool_default_size = 2  # Write an object n times.

osd_pool_default_min_size = 1 # Allow writing n copy in a degraded state.

osd_pool_default_pg_num = 672

osd_pool_default_pgp_num = 672

osd_crush_chooseleaf_type = 1

mon_osd_full_ratio = .75

mon_osd_nearfull_ratio = .65

osd_backfill_full_ratio = .65

mon_clock_drift_allowed = .15

mon_clock_drift_warn_backoff = 30

mon_osd_down_out_interval = 300

mon_osd_report_timeout = 300

filestore_xattr_use_omap = true

 

 

Any assistance would be greatly appreciated!

 

Cheers,

 

Ben

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux