Re: Performance problems

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



There is one SSD in each node.  IPoIB performance is about 7 gbps between each host.  CephFS is mounted via kernel client.  Ceph version is ceph-0.56.3-1.  I have a 1GB journal on the same drive as the OSD but on a seperate file system split via LVM.  

Here is output of another test with fdatasync:

[root@triton temp]# dd conv=fdatasync if=/dev/zero of=/mnt/temp/test.out bs=512k count=10000
10000+0 records in
10000+0 records out
5242880000 bytes (5.2 GB) copied, 359.307 s, 14.6 MB/s
[root@triton temp]# dd if=/mnt/temp/test.out of=/dev/null bs=512k count=10000
10000+0 records in
10000+0 records out
5242880000 bytes (5.2 GB) copied, 14.0521 s, 373 MB/s

The network traffic appears to match the transfer speeds shown here too.  Writing is very slow.


On Mon, Apr 8, 2013 at 3:04 PM, Mark Nelson <mark.nelson@xxxxxxxxxxx> wrote:
Hi,

How many drives?  Have you tested your IPoIB performance with iperf?  Is this CephFS with the kernel client?  What version of Ceph?  How are your journals configured? etc.  It's tough to make any recommendations without knowing more about what you are doing.

Also, please use conv=fdatasync when doing buffered IO writes with dd.

Thanks,
Mark


On 04/08/2013 03:00 PM, Ziemowit Pierzycki wrote:
Hi,

The first test was writing 500 mb file and was clocked at 1.2 GBps.  The
second test was writing 5000 mb file at 17 MBps.  The third test was
reading the file at ~400 MBps.


On Mon, Apr 8, 2013 at 2:56 PM, Gregory Farnum <greg@xxxxxxxxxxx
<mailto:greg@xxxxxxxxxxx>> wrote:

    More details, please. You ran the same test twice and performance went
    up from 17.5MB/s to 394MB/s? How many drives in each node, and of what
    kind?
    -Greg
    Software Engineer #42 @ http://inktank.com | http://ceph.com


    On Mon, Apr 8, 2013 at 12:38 PM, Ziemowit Pierzycki
    <ziemowit@xxxxxxxxxxxxx <mailto:ziemowit@xxxxxxxxxxxxx>> wrote:
     > Hi,
     >
     > I have a 3 node SSD-backed cluster connected over infiniband (16K
    MTU) and
     > here is the performance I am seeing:
     >
     > [root@triton temp]# !dd
     > dd if=/dev/zero of=/mnt/temp/test.out bs=512k count=1000
     > 1000+0 records in
     > 1000+0 records out
     > 524288000 bytes (524 MB) copied, 0.436249 s, 1.2 GB/s
     > [root@triton temp]# dd if=/dev/zero of=/mnt/temp/test.out bs=512k
     > count=10000
     > 10000+0 records in
     > 10000+0 records out
     > 5242880000 bytes (5.2 GB) copied, 299.077 s, 17.5 MB/s
     > [root@triton temp]# dd if=/mnt/temp/test.out of=/dev/null bs=512k
     > count=1000010000+0 records in
     > 10000+0 records out
     > 5242880000 bytes (5.2 GB) copied, 13.3015 s, 394 MB/s
     >
     > Does that look right?  How do I check this is not a network
    problem, because
     > I remember seeing a kernel issue related to large MTU.
     >
     > _______________________________________________
     > ceph-users mailing list
     > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxx.com>
     > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

     >




_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux