On small cluster i've get a great sequental perfomance by using btrfs on OSD, journal file (max sync interval ~180s) and with option filestore journal parallel = true 2015-11-11 10:12 GMT+03:00 Ben Town <ben@xxxxxxxxxxxxxxxxxxxx>: > Hi Guys, > > > > I’m in the process of configuring a ceph cluster and am getting some less > than ideal performance and need some help figuring it out! > > > > This cluster will only really be used for backup storage for Veeam so I > don’t need a crazy amount of I/O but good sequential writes would be ideal. > > > > At the moment I am only getting about 50MBs according to rados bench. > > > > Our environment looks like this. > > > > 3x OSD Servers: > > · 1x E3 CPU > > · 24 GB RAM > > · 8x 3TB Enterprise Drives > > · 2x SSDs for journals – I haven’t used enterprise ones, just > whatever I had around the office. I figure that this could be the bottle > neck but I’d figure I’d get more than 50MBs. > > · 1x 2 port 10GB NIC with separate public / replication networks > > > > 3x Monitor Servers > > · 2x Quad > > · 16GB RAM > > · 2x 300GB SAS RAID 1 > > · 1x 2 port 10GB NIC > > > > My results from bench tests are: > > > > DD on the RBD device is showing ~50-60MBs > > DD on the disks is showing ~160MBs > > Rados Bench is showing ~50MBs > > Network is showing 9.6Gbits on all servers > > FIO is showing the below: > > > > WRITE: io=1024.0MB, aggrb=3099KB/s, minb=3099KB/s, maxb=3099KB/s, > mint=338300msec, maxt=338300msec > > > > > > My ceph config is as follows: > > > > mon_host = 192.168.78.101,192.168.78.102,192.168.78.103 > > public_network = 192.168.78.0/24 > > cluster_network = 192.168.79.0/24 > > ms_bind_ipv6 = false > > max_open_files = 131072 > > auth_cluster_required = cephx > > auth_service_required = cephx > > auth_client_required = cephx > > osd_journal_size = 10000 > > osd_pool_default_size = 2 # Write an object n times. > > osd_pool_default_min_size = 1 # Allow writing n copy in a degraded state. > > osd_pool_default_pg_num = 672 > > osd_pool_default_pgp_num = 672 > > osd_crush_chooseleaf_type = 1 > > mon_osd_full_ratio = .75 > > mon_osd_nearfull_ratio = .65 > > osd_backfill_full_ratio = .65 > > mon_clock_drift_allowed = .15 > > mon_clock_drift_warn_backoff = 30 > > mon_osd_down_out_interval = 300 > > mon_osd_report_timeout = 300 > > filestore_xattr_use_omap = true > > > > > > Any assistance would be greatly appreciated! > > > > Cheers, > > > > Ben > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Have a nice day, Timofey. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |