On Fri, 24 Sep 2010, DongJin Lee wrote: > > Wow, these levels seem oddly slow. What filesystem are those disks running? > > What tool are you using to test IOPS, and what's your network setup? > > If you can move the journal onto a separate device it will help. > > -Greg > > > > I have changed the OSD to SSD (OCZ Vertex2) > To simplify the problem, only one OSD is used, > so now, 3 PCs - 1OSD, 1MDS, 1MON (all similar spec, except the OSD's > 1TB disk replaced with 60GB SSD), using btrfs > all connected to 1G/s managed switch. Iperf gives about 980Mb/s > > I ran dbench, on the OSD PC. > Local vs /ceph > > using dbench -t 10 100 -D /data/osd0/ > - gives 150MB/s, good and as expected. > > using dbench -t 10 100 -D /ceph > - gives 15MB/s The dbench workload is metadata intensive, and most metadata (update) operations require a round trip to the MDS. The client/mds protocol is pretty aggressive about issueing leases to the client, but that only goes so far. Metadata performance will never be as fast as a local fs for a single process. > Also, I often get errors from dbench while benchmarking (only when > /ceph mounts, not elsewhere) > > [323] unlink '/media/cephlocal/clients/client25/~dmtmp/WORD/CHAP10.DOC' > failed - No such file or directory > 52 cleanup 14 sec > 39 cleanup 15 sec > [323] unlink '/media/cephlocal/clients/client24/~dmtmp/WORDPRO/NEWS1_1.LWP' > failed - No such file or directory > [323] unlink '/media/cephlocal/clients/client91/~dmtmp/WORD/CHAP10.DOC' > failed - No such file or directory > [323] unlink '/media/cephlocal/clients/client27/~dmtmp/WORDPRO/RESULTS.XLS' > failed - No such file or directory > > and > > [31] open /media/cephlocal/clients/client258/filler.001 failed for > handle 9939 (No such file or directory) > [31] open /media/cephlocal/clients/client220/filler.001 failed for > handle 9939 (No such file or directory) > [111] open /media/cephlocal/clients/client218/~dmtmp/WORD failed for > handle 9943 (No such file or directory) > [91] open /media/cephlocal/clients/client986/filler.004 failed for > handle 9942 (No such file or directory) What version of the client are you running? I was hitting this (or a similar) problem with the dbench workload a couple weeks ago and it turned out to be a long standing bug in fs/namei.c that has finally (finally!) been fixed in 2.6.36-rc2 (2e2e88ea). I hadn't seen it actually come up for over a year, so something must have changed with the kclient or MDS recently that made the bug happen, but in any case it's fixed now. sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
