Hello, On Wed, 11 Nov 2015 07:12:56 +0000 Ben Town wrote: > Hi Guys, > > I'm in the process of configuring a ceph cluster and am getting some > less than ideal performance and need some help figuring it out! > > This cluster will only really be used for backup storage for Veeam so I > don't need a crazy amount of I/O but good sequential writes would be > ideal. > > At the moment I am only getting about 50MBs according to rados bench. > > Our environment looks like this. > > 3x OSD Servers: > > * 1x E3 CPU > Quite non-specific. > * 24 GB RAM > Bordering on too little, but not dangerously so. > * 8x 3TB Enterprise Drives > > * 2x SSDs for journals - I haven't used enterprise ones, just > whatever I had around the office. I figure that this could be the bottle > neck but I'd figure I'd get more than 50MBs. > If you peruse the archives of this ML you will threads where exactly this was the case and also ways to verify if those SSDs are usable or not. Citing the exact models might have resulted in more specific replies... Watch your storage nodes with atop or similar, you should spot the bottleneck (most likely the SSDs) readily. Christian > * 1x 2 port 10GB NIC with separate public / replication networks > > 3x Monitor Servers > > * 2x Quad > > * 16GB RAM > > * 2x 300GB SAS RAID 1 > > * 1x 2 port 10GB NIC > > My results from bench tests are: > > DD on the RBD device is showing ~50-60MBs > DD on the disks is showing ~160MBs > Rados Bench is showing ~50MBs > Network is showing 9.6Gbits on all servers > FIO is showing the below: > > WRITE: io=1024.0MB, aggrb=3099KB/s, minb=3099KB/s, maxb=3099KB/s, > mint=338300msec, maxt=338300msec > > > My ceph config is as follows: > > mon_host = 192.168.78.101,192.168.78.102,192.168.78.103 > public_network = 192.168.78.0/24 > cluster_network = 192.168.79.0/24 > ms_bind_ipv6 = false > max_open_files = 131072 > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > osd_journal_size = 10000 > osd_pool_default_size = 2 # Write an object n times. > osd_pool_default_min_size = 1 # Allow writing n copy in a degraded state. > osd_pool_default_pg_num = 672 > osd_pool_default_pgp_num = 672 > osd_crush_chooseleaf_type = 1 > mon_osd_full_ratio = .75 > mon_osd_nearfull_ratio = .65 > osd_backfill_full_ratio = .65 > mon_clock_drift_allowed = .15 > mon_clock_drift_warn_backoff = 30 > mon_osd_down_out_interval = 300 > mon_osd_report_timeout = 300 > filestore_xattr_use_omap = true > > > Any assistance would be greatly appreciated! > > Cheers, > > Ben -- Christian Balzer Network/Systems Engineer chibi@xxxxxxx Global OnLine Japan/Fusion Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |