|
I have a performance problem i would like
advise. I have the following sub-optimal setup: * 2 Servers (WTFM008 WTFM009) * HP Proliant DL180 * SmartArray G6 P410 raid-controller * 4x 500GB RAID5 (seq writes = 230MB/s) * CentOS 6.5 x86_64 * 2.000.000 files (ms-word), with no directory structure * Ceph * ceph-deploy mon create WTFM008 WTFM009 * ceph-deploy mds create WTFM008 WTFM009 * ceph-deploy osd activate WTFM008:/var/lib/ceph/osd/ceph-0 WTFM009:/var/lib/ceph/osd/ceph-1 (osd is using root fs) * ceph-fuse /mnt/ceph I am currently trying to copy 2 million ms-word documents into ceph. When i started it was doing about 10 files per second. Now, 1 week later, it has done about 500.000 files and has slowed down to 1 file per 10 seconds. How can i improve this terrible performance? * The hardware is a fixed configuration, i cannot add (SSD) disks or change RAID. * I could not find the cephfs kernel module so i had to use cephfs-fuse. * I could have started with a degraded setup (1 OSD) for the initial load, would that have helped in the performance? (Ceph not having to do the distribution part) * There is nu load on the systems at all (not cpu, not mem, not disk i/o) Below is my crush map. Regards, Samuel Terburg Panther-IT BV # begin crush map # devices device 0 osd.0 device 1 osd.1 # types type 0 osd type 1 host type 2 rack type 3 row type 4 room type 5 datacenter type 6 root # buckets host WTFM008 { id -2 # do not change unnecessarily # weight 1.340 alg straw hash 0 # rjenkins1 item osd.0 weight 1.340 } host WTFM009 { id -3 # do not change unnecessarily # weight 1.340 alg straw hash 0 # rjenkins1 item osd.1 weight 1.340 } root default { id -1 # do not change unnecessarily # weight 2.680 alg straw hash 0 # rjenkins1 item WTFM008 weight 1.340 item WTFM009 weight 1.340 } # rules rule data { ruleset 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } rule metadata { ruleset 1 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } rule rbd { ruleset 2 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } # end crush map # ceph -w cluster 4f7bcb26-0cee-4472-abca-c200a999b686 health HEALTH_OK monmap e1: 2 mons at {WTFM008=192.168.0.1:6789/0,WTFM009=192.168.0.2:6789/0}, election epoch 4, quorum 0,1 WTFM008,WTFM009 mdsmap e5: 1/1/1 up {0=WTFM008=up:active}, 1 up:standby osdmap e14: 2 osds: 2 up, 2 in pgmap v151668: 192 pgs, 3 pools, 31616 MB data, 956 kobjects 913 GB used, 1686 GB / 2738 GB avail 192 active+clean client io 40892 kB/s rd, 7370 B/s wr, 1 op/s |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
