Hi, can you post your ceph.conf ? Which tools do you use for benchmark ? which block size, iodepth, number of client/rbd volume do you use ? Is it with krbd kernel driver ? (I have seen some bad performance with kernel 3.16, but at much higher rate (100k iops) Is it with ethernet switches ? or ip over infiniband ? your results seem quite low anyway. I'm also using ethernet mellanox switchs (10GBE), sas3008 (dell r630). and I can reach around 250kiops randread 4K with 1osd (with 80% usage of 2x10cores 3,1ghz) here my ceph.conf ----------------- [global] fsid = .... public_network =.... mon_initial_members = ... mon_host =..... auth_cluster_required = none auth_service_required = none auth_client_required = none filestore_xattr_use_omap = true osd_pool_default_min_size = 1 debug_lockdep = 0/0 debug_context = 0/0 debug_crush = 0/0 debug_buffer = 0/0 debug_timer = 0/0 debug_journaler = 0/0 debug_osd = 0/0 debug_optracker = 0/0 debug_objclass = 0/0 debug_filestore = 0/0 debug_journal = 0/0 debug_ms = 0/0 debug_monc = 0/0 debug_tp = 0/0 debug_auth = 0/0 debug_finisher = 0/0 debug_heartbeatmap = 0/0 debug_perfcounter = 0/0 debug_asok = 0/0 debug_throttle = 0/0 osd_op_threads = 5 filestore_op_threads = 4 osd_op_num_threads_per_shard = 2 osd_op_num_shards = 10 filestore_fd_cache_size = 64 filestore_fd_cache_shards = 32 ms_nocrc = true ms_dispatch_throttle_bytes = 0 cephx_sign_messages = false cephx_require_signatures = false throttler_perf_counter = false ms_crc_header = false ms_crc_data = false [osd] osd_client_message_size_cap = 0 osd_client_message_cap = 0 osd_enable_op_tracker = false (main boost are disable cephx_auth, debug, and increase thread sharding) ----- Mail original ----- De: "Jacek Jarosiewicz" <jjarosiewicz@xxxxxxxxxxxxx> À: "ceph-users" <ceph-users@xxxxxxxxxxxxxx> Envoyé: Mercredi 17 Juin 2015 11:10:26 Objet: rbd performance issue - can't find bottleneck Hi, We've been doing some testing of ceph hammer (0.94.2), but the performance is very slow and we can't find what's causing the problem. Initially we've started with four nodes with 10 osd's total. The drives we've used were SATA enterprise drives and on top of that we've used SSD drives as flashcache devices for SATA drives and for storing OSD's journal. The local tests on each of the four nodes are giving the results you'd expect: ~500MB/s seq writes and reads from SSD's, ~40k iops random reads from SSD's, ~200MB/s seq writes and reads from SATA drives ~600 iops random reads from SATA drives ..but when we've tested this setup from a client we got rather slow results.. so we've tried to find a bottleneck and tested the network by connecting client to our nodes via NFS - and performance via NFS is as expected (similar results to local tests, only slightly slower). So we've reconfigured ceph to not use SATA drives and just setup OSD's on SSD drives (we wanted to test if maybe this is a flashcache problem?) ..but to no success, the results of rbd i/o tests from two osd nodes setup on SSD drives are like this: ~60MB/s seq writes ~100MB/s seq reads ~2-3k iops random reads The client is an rbd mounted on a linux ubuntu box. All the servers (osd nodes and the client) are running Ubuntu Server 14.04. We tried to switch to CentOS 7 - but the results are the same. Here are some technical details about our setup: Four exact same osd nodes: E5-1630 CPU 32 GB RAM Mellanox MT27520 56Gbps network cards SATA controller LSI Logic SAS3008 Storage nodes are connected to SuperMicro chassis: 847E1C-R1K28JBOD Four monitors (one on each node). We do not use CephFS so we do not run ceph-mds. During the tests we were monitoring all osd nodes and the client - we haven't seen any problems on none of the hosts - load was low, there were no cpu waits, no abnormal system interrupts, no i/o problems on the disks - all the systems seemed to not sweat at all and yet the results are rather dissatisfying.. we're kinda lost, any help will be appreciated. Cheers, J -- Jacek Jarosiewicz Administrator Systemów Informatycznych ---------------------------------------------------------------------------------------- SUPERMEDIA Sp. z o.o. z siedzibą w Warszawie ul. Senatorska 13/15, 00-075 Warszawa Sąd Rejonowy dla m.st.Warszawy, XII Wydział Gospodarczy Krajowego Rejestru Sądowego, nr KRS 0000029537; kapitał zakładowy 42.756.000 zł NIP: 957-05-49-503 Adres korespondencyjny: ul. Jubilerska 10, 04-190 Warszawa ---------------------------------------------------------------------------------------- SUPERMEDIA -> http://www.supermedia.pl dostep do internetu - hosting - kolokacja - lacza - telefonia _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |