Hey, guys.
I'm still feeling unlucky about these experiments. Here's what I did:
1) Set
the parameters described below in ceph.conf
2) Push
the ceph.conf to all nodes using ceph-deploy
3) Restart
monitor, mds and osd’s on all nodes
4) Ran
the test twice atleast and look at the results which came out the second time.
5) I
mounted the filesystem using mount –t ceph 10.76.38.57:/ /mnt/mycephfs –o
name=admin,secret=xxxxxxxxx
6) Ran the benchmark
I modified the following parameters and ran each test
separately
-
“osd journal size” to 5 GB and 10 GB and 20 GB
-
“osd client message size cap” to 0,1,1024
-
“osd pool default min size” to 1 and 3
In all the above I observed similar pattern as below.
-
about 5-6 Gbps throughput at the start of the test and gradually
dropping to 900 Mbps till the test completes. I also observed that post 150-160
files being written there is a wait for about 10-15 seconds before the next
file is written.
Test using FUSE.
I also installed FUSE on cephnode1 and mounted the fuse
mount using the following command
ceph-fuse –m 10.76.38.56 /mnt/mycephfs
I saw a drastic reduction in write throughput to around 170
Mbps. The system took about 5-6 seconds before it started writing any files and
was constantly at 150 – 180 Mbps write through put when the directory was mounted
using FUSE.
Any additional thoughts? Would the problem be due to my NFS client?
Regards,
F.
From: Gregory Farnum <gfarnum@xxxxxxxxxx>
To: Patrick Donnelly <pdonnell@xxxxxxxxxx>
Cc: Fabiano de O. Lucchese <flucchese@xxxxxxxxx>; "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx>
Sent: Tuesday, July 19, 2016 5:23 PM
Subject: Re: CephFS write performance
On Tue, Jul 19, 2016 at 9:39 AM, Patrick Donnelly <pdonnell@xxxxxxxxxx> wrote:
> On Tue, Jul 19, 2016 at 10:25 AM, Fabiano de O. Lucchese
> <flucchese@xxxxxxxxx> wrote:
>> I configured the cluster to replicate data twice (3 copies), so these
>> numbers fall within my expectations. So far so good, but here's comes the
>> issue: I configured CephFS and mounted a share locally on one of my servers.
>> When I write data to it, it shows abnormally high performance at the
>> beginning for about 5 seconds, stalls for about 20 seconds and then picks up
>> again. For long running tests, the observed write throughput is very close
>> to what the rados bench provided (about 640 MB/s), but for short-lived
>> tests, I get peak performances of over 5GB/s. I know that journaling is
>> expected to cause spiky performance patters like that, but not to this
>> level, which makes me think that CephFS is buffering my writes and returning
>> the control back to client before persisting them to the jounal, which looks
>> undesirable.
>
> The client is buffering the writes to RADOS which would give you the
> abnormally high initial performance until the cache needs flushed. You
> might try tweaking certain osd settings:
>
> http://docs.ceph.com/docs/hammer/rados/configuration/osd-config-ref/
>
> in particular: "osd client message size cap". Also:
I am reasonably sure you don't want to change the message size cap;
that's entirely an OSD-side throttle about how much dirty data it lets
in before it stops reading off the wire — and I don't think the client
feeds back from outgoing data. More likely it's about how much dirty
data is being absorbed by the Client before it forces writes out to
the OSDs and you want to look at
client_oc_size (default 1024*1024*200, aka 200MB)
client_oc_max_dirty (default 100MB)
client_oc_target_dirty (default 8MB)
and turn down the max dirty limits if you're finding it's too bumpy a ride.
-Greg
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
