Ceph performance

Frédéric HILPERT <frederic@xxxxxxxxxx> · Mon, 22 Apr 2013 11:46:32 +0200



    Hi, 

    
    I have a few questions about performance before putting ceph in
    production. I have a test lab with 3 hosts and 2 OSD each (on
    dedicated disks)

    The disks have about 100MB/sec write throughput. 

    
    I noticed that when the journal is on the OSD itself, the write
    throughput is only around 35MB/sec because there are two
    simultaneous writes on the disk. With journal on SSD I have
    100MB/sec and more write speed.

    
    I use block devices so, I don't have any mds.

    
    My questions : 

    
      Is there a way to have a good disk bandwidth without using SSD
        for journal ?
      What are your recommendations ? 
      Should mds help in improving this write throughput ?
        (Remember, I am using block devices)
    
    Here are my test lab results : 

    
    fred@ceph1:~$ ceph osd tree

      
      # id    weight    type name    up/down    reweight

      -1    6    root default

      -3    6        rack unknownrack

      -2    2            host ceph1

      0    1                osd.0    up    1    

      1    1                osd.1    up    1    

      -4    2            host ceph2

      2    1                osd.2    up    1    

      3    1                osd.3    up    1    

      -5    2            host ceph3

      4    1                osd.4    up    1    

      5    1                osd.5    up    1    

    
      host ceph1 is setup with a disk for system and two dedicated
        HDD for OSD 

      
      host ceph3 is setup with a disk for system, SSD for journals
        and two dedicated HDD for OSD
    
    Let's bench : 

    
    2013-04-22 11:41:21.843396 osd.4 [INF] bench: wrote 1024 MB in
      blocks of 4096 KB in 2.629238 sec at 389 MB/sec

      2013-04-22 11:41:26.536869 osd.5 [INF] bench: wrote 1024 MB in
      blocks of 4096 KB in 4.194540 sec at 244 MB/sec

      2013-04-22 11:41:43.758835 osd.0 [INF] bench: wrote 1024 MB in
      blocks of 4096 KB in 29.455098 sec at 35599 KB/sec

      2013-04-22 11:41:46.278314 osd.1 [INF] bench: wrote 1024 MB in
      blocks of 4096 KB in 29.944204 sec at 35017 KB/sec

    
    We can see that the host with HDD journal is working really slow,
      and the host with SSD journal really fast --> Faster than the
      HDD themselves ! 

      Besides, do the SSD journal work as SSD cache ?

    
    Best Regards,

      Fred

    
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com