On 29.10.2012 22:57, Sam Lang wrote:
Hi Roman,
Is this with the ceph fuse client or the ceph kernel module?
Its not surprising that the local file system (/home) is so much
faster than a mounted ceph volume, especially the first time the
directory tree is traversed (metadata results are cached at the client
to improve performance). Try running the same find command on the
ceph volume and see if the cached results at the client improve
performance at all.
In order to understand what the performance of ceph should be capable
of doing with your deployment for this specific workload, you should
run iperf between two nodes to get an idea of your latency limits.
Also, I noticed that the real timings you listed for ceph and /home
are offset by exactly 17 minutes (user and sys are identical). Was
that a copy/paste error, by chance?
-sam
On 10/29/2012 09:01 AM, Roman Alekseev wrote:
Hi,
Kindly guide me how to improve performance on the cluster which consist
of 5 dedicated servers:
- ceph.conf: http://pastebin.com/hT3qEhUF
- file system on all drives is ext4
- mount options "user_xattr"
- each server has :
CPU:Intel® Xeon® Processor E5335(8M Cache, 2.00 GHz, 1333 MHz FSB) x2
MEM: 4Gb DDR2
- 1Gb network
Simple test:
mounted as ceph
root@client1:/mnt/mycephfs# time find . | wc -l
83932
real 17m55.399s
user 0m0.152s
sys 0m1.528s
on 1 HDD:
root@client1:/home# time find . | wc -l
83932
real 0m55.399s
user 0m0.152s
sys 0m1.528s
Please help me to find out the issue. Thanks.
Hi Sam,
I use the Ceph fs only as kernel module, because we need to get its
powerful performance but as I can see it is slower then distributed file
system based on fuse, for example, MooseFS performed the same test for 3
min.
Here is the result iperf test beetwen client and osd server:
root@asrv151:~# iperf -c client -i 1
------------------------------------------------------------
Client connecting to clientIP, TCP port 5001
TCP window size: 96.1 KByte (default)
------------------------------------------------------------
[ 3] local osd_server port 50106 connected with clientIP port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 112 MBytes 941 Mbits/sec
[ 3] 1.0- 2.0 sec 110 MBytes 924 Mbits/sec
[ 3] 2.0- 3.0 sec 108 MBytes 905 Mbits/sec
[ 3] 3.0- 4.0 sec 109 MBytes 917 Mbits/sec
[ 3] 4.0- 5.0 sec 110 MBytes 926 Mbits/sec
[ 3] 5.0- 6.0 sec 109 MBytes 915 Mbits/sec
[ 3] 6.0- 7.0 sec 110 MBytes 926 Mbits/sec
[ 3] 7.0- 8.0 sec 108 MBytes 908 Mbits/sec
[ 3] 8.0- 9.0 sec 107 MBytes 897 Mbits/sec
[ 3] 9.0-10.0 sec 106 MBytes 886 Mbits/sec
[ 3] 0.0-10.0 sec 1.06 GBytes 914 Mbits/sec
ceph -w results:
health HEALTH_OK
monmap e3: 3 mons at {a=mon.a:6789/0,b=mon.b:6789/0,c=mon.c:6789/0},
election epoch 10, quorum 0,1,2 a,b,c
osdmap e132: 5 osds: 5 up, 5 in
pgmap v11720: 384 pgs: 384 active+clean; 1880 MB data, 10679 MB
used, 5185 GB / 5473 GB avail
mdsmap e4: 1/1/1 up {0=a=up:active}
2012-10-30 12:23:09.830677 osd.2 [WRN] slow request 30.135787 seconds
old, received at 2012-10-30 12:22:39.694780: osd_op(mds.0.1:309216
10000017163.00000000 [setxattr path (69),setxattr parent (196),tmapput
0~596] 1.724c80f7) v4 currently waiting for sub ops
2012-10-30 12:23:10.109637 mon.0 [INF] pgmap v11720: 384 pgs: 384
active+clean; 1880 MB data, 10679 MB used, 5185 GB / 5473 GB avail
2012-10-30 12:23:12.918038 mon.0 [INF] pgmap v11721: 384 pgs: 384
active+clean; 1880 MB data, 10680 MB used, 5185 GB / 5473 GB avail
2012-10-30 12:23:13.977044 mon.0 [INF] pgmap v11722: 384 pgs: 384
active+clean; 1880 MB data, 10681 MB used, 5185 GB / 5473 GB avail
2012-10-30 12:23:10.587391 osd.3 [WRN] 6 slow requests, 6 included
below; oldest blocked for > 30.808352 secs
2012-10-30 12:23:10.587398 osd.3 [WRN] slow request 30.808352 seconds
old, received at 2012-10-30 12:22:39.778971: osd_op(mds.0.1:308701
200.000002e5 [write 976010~5402] 1.adbeb1a) v4 currently waiting for sub ops
2012-10-30 12:23:10.587403 osd.3 [WRN] slow request 30.796417 seconds
old, received at 2012-10-30 12:22:39.790906: osd_op(mds.0.1:308702
200.000002e5 [write 981412~6019] 1.adbeb1a) v4 currently waiting for sub ops
2012-10-30 12:23:10.587408 osd.3 [WRN] slow request 30.796347 seconds
old, received at 2012-10-30 12:22:39.790976: osd_op(mds.0.1:308703
200.000002e5 [write 987431~61892] 1.adbeb1a) v4 currently waiting for
sub ops
2012-10-30 12:23:10.587413 osd.3 [WRN] slow request 30.530228 seconds
old, received at 2012-10-30 12:22:40.057095: osd_op(mds.0.1:308704
200.000002e5 [write 1049323~6630] 1.adbeb1a) v4 currently waiting for
sub ops
2012-10-30 12:23:10.587417 osd.3 [WRN] slow request 30.530027 seconds
old, received at 2012-10-30 12:22:40.057296: osd_op(mds.0.1:308705
200.000002e5 [write 1055953~20679] 1.adbeb1a) v4 currently waiting for
sub ops
At the same time I'm copy data to ceph mounted storage.
I dunno what can I do to resolve this problem :(
Any advices will be greatly appreciated.
--
Kind regards,
R. Alekseev
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
