Hi! Seeking performance improvement in our cluster (Firefly 0.80.7 on Wheezy, 5 nodes, 58 osds), I wrote a small python script, that walks through ceph nodes and issue 'perf dump' command on osd admin sockets. It extracts *_latency tuples, calculate min/max/avg, compare osd perf metrics with min/avg of whole cluster or same host and display result in table form. The goal - to check where the most latency is. The hardware is not new and shiny: - 5 nodes * 10-12 OSDs each - Intel E5520@2.26/32-48Gb DDR3-1066 ECC - 10Gbit X520DA interconnect - Intel DC3700 200Gb as a system volume + journals, connected to sata2 onboard in ahci mode - Intel RS2MB044 / RS2BL080 SAS RAID in RAID0 per drive mode, WT, disk cache disabled - bunch of 1Tb or 2Tb various WD Black drives, 58 disks, 76Tb total - replication = 3, filestore on xfs - shared client and cluster 10Gbit network - cluster used as rbd storage for VMs - rbd_cache is on by 'cache=writeback' in libvirt (I suppose, that it is true ;)) - no special tuning in ceph.conf: >osd mount options xfs = rw,noatime,inode64 >osd disk threads = 2 >osd op threads = 8 >osd max backfills = 2 >osd recovery max active = 2 I get rather slow read performance from within VM, especially with QD=1, so many VMs are running slowly. I think, that this HW config can perform better, as I got 10-12k iops with QD=32 from time to time. So I have some questions: 1. Am I right, that osd perfs are cumulative and counting up from OSD start? 2. Is any way to reset perf counters without restating OSD daemon? Maybe a command through admin socket? 3. What latencies should I expect from my config, or, what latencies you have on yours clusters? Just an example or as a reference to compare with my values. I've interesting mostly in - 'op_latency', - 'op_[r|w]_latency', - 'op_[r|w]_process_latency' - 'journal_latency' But other parameters, like 'apply_latency' or 'queue_transaction_latency_avg' are also interesting to compare. 4. Where I have to look firstly, if I need to improve QD=1 (i. e. per-VM) performance. Thanks! Megov Igor CIO, Yuterra _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |