Hi All
I've just installed Jewel 10.2.2 on hardware that has previously been running Giant. Rados Bench with the default rand and seq tests is giving me approx 40% of the throughput I used to achieve. On Giant I would get ~1000MB/s (so probably limited by the 10GbE interface), now I'm getting 300 - 400MB/s.
I can see there is no activity on the disks during the bench so the data is all coming out of cache. The cluster isn't doing anything else during the test. I'm fairly sure my network is sound, I've done the usual testing with iperf etc. The write test seems about the same as I used to get (~400MB/s).
This was a fresh install rather than an upgrade.
Are there any gotchas I should be aware of?
Some more details:
OS: CentOS 7
Kernel: 3.10.0-327.28.2.el7.x86_64
5 nodes (each 10 * 4TB SATA, 2 * Intel dc3700 SSD partitioned up for journals).
10GbE public network
10GbE cluster network
MTU 9000 on all interfaces and switch
Ceph installed from ceph repo
Ceph.conf is pretty basic (IPs, hosts etc omitted):
filestore_xattr_use_omap = true
osd_journal_size = 10000
osd_pool_default_size = 3
osd_pool_default_min_size = 2
osd_pool_default_pg_num = 4096
osd_pool_default_pgp_num = 4096
osd_crush_chooseleaf_type = 1
max_open_files = 131072
mon_clock_drift_allowed = .15
mon_clock_drift_warn_backoff = 30
mon_osd_down_out_interval = 300
mon_osd_report_timeout = 300
mon_osd_full_ratio = .95
mon_osd_nearfull_ratio = .80
osd_backfill_full_ratio = .80
Thanks
David
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
