Re: Performance issues on small cluster

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On small cluster i've get a great sequental perfomance by using btrfs
on OSD, journal file (max sync interval ~180s) and with option
filestore journal parallel = true

2015-11-11 10:12 GMT+03:00 Ben Town <ben@xxxxxxxxxxxxxxxxxxxx>:
> Hi Guys,
>
>
>
> I’m in the process of configuring a ceph cluster and am getting some less
> than ideal performance and need some help figuring it out!
>
>
>
> This cluster will only really be used for backup storage for Veeam so I
> don’t need a crazy amount of I/O but good sequential writes would be ideal.
>
>
>
> At the moment I am only getting about 50MBs according to rados bench.
>
>
>
> Our environment looks like this.
>
>
>
> 3x OSD Servers:
>
> ·         1x E3 CPU
>
> ·         24 GB RAM
>
> ·         8x 3TB Enterprise Drives
>
> ·         2x SSDs for journals – I haven’t used enterprise ones, just
> whatever I had around the office. I figure that this could be the bottle
> neck but I’d figure I’d get more than 50MBs.
>
> ·         1x 2 port 10GB NIC with separate public / replication networks
>
>
>
> 3x Monitor Servers
>
> ·         2x Quad
>
> ·         16GB RAM
>
> ·         2x 300GB SAS RAID 1
>
> ·         1x 2 port 10GB NIC
>
>
>
> My results from bench tests are:
>
>
>
> DD on the RBD device is showing ~50-60MBs
>
> DD on the disks is showing ~160MBs
>
> Rados Bench is showing ~50MBs
>
> Network is showing 9.6Gbits on all servers
>
> FIO is showing the below:
>
>
>
>   WRITE: io=1024.0MB, aggrb=3099KB/s, minb=3099KB/s, maxb=3099KB/s,
> mint=338300msec, maxt=338300msec
>
>
>
>
>
> My ceph config is as follows:
>
>
>
> mon_host = 192.168.78.101,192.168.78.102,192.168.78.103
>
> public_network = 192.168.78.0/24
>
> cluster_network = 192.168.79.0/24
>
> ms_bind_ipv6 = false
>
> max_open_files = 131072
>
> auth_cluster_required = cephx
>
> auth_service_required = cephx
>
> auth_client_required = cephx
>
> osd_journal_size = 10000
>
> osd_pool_default_size = 2  # Write an object n times.
>
> osd_pool_default_min_size = 1 # Allow writing n copy in a degraded state.
>
> osd_pool_default_pg_num = 672
>
> osd_pool_default_pgp_num = 672
>
> osd_crush_chooseleaf_type = 1
>
> mon_osd_full_ratio = .75
>
> mon_osd_nearfull_ratio = .65
>
> osd_backfill_full_ratio = .65
>
> mon_clock_drift_allowed = .15
>
> mon_clock_drift_warn_backoff = 30
>
> mon_osd_down_out_interval = 300
>
> mon_osd_report_timeout = 300
>
> filestore_xattr_use_omap = true
>
>
>
>
>
> Any assistance would be greatly appreciated!
>
>
>
> Cheers,
>
>
>
> Ben
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Have a nice day,
Timofey.
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux