On Fri, Oct 2, 2015 at 1:57 AM, John Spray <jspray@xxxxxxxxxx> wrote: > On Fri, Oct 2, 2015 at 2:42 AM, Goncalo Borges > <goncalo@xxxxxxxxxxxxxxxxxxx> wrote: >> Dear CephFS Gurus... >> >> I have a question regarding ceph-fuse and its memory usage. >> >> 1./ My Ceph and CephFS setups are the following: >> >> Ceph: >> a. ceph 9.0.3 >> b. 32 OSDs distributed in 4 servers (8 OSD per server). >> c. 'osd pool default size = 3' and 'osd pool default min size = 2' >> d. All servers running Centos6.7 >> >> CephFS: >> e. a single mds >> f. dedicated pools for data and metadata >> g. clients in different locations / sites mounting CephFS via ceph-fuse >> h. All servers and clients running Centos6.7 >> >> 2./ I have been running fio tests in two CephFS clients: >> - Client A is in the same data center as all OSDs connected at 1 GbE >> - Client B is in a different data center (in another city) also >> connected at 1 GbE. However, I've seen that the connection is problematic, >> and sometimes, the network performance is well bellow the theoretical 1 Gbps >> limit. >> - Client A has 24 GB RAM + 98 GB of SWAP and client B has 48 GB of RAM + >> 98 GB of SWAP >> >> 3./ I have been running some fio write tests (with 128 threads) in both >> clients, and surprisingly, the results show that the aggregated throughput >> is better for client B than client A. >> >> CLIENT A results: >> # grep agg >> fio128threadsALL/fio128write_ioenginelibaio_iodepth64_direct1_bs512K_20151001015558.out >> WRITE: io=1024.0GB, aggrb=114878KB/s, minb=897KB/s, maxb=1785KB/s, >> mint=4697347msec, maxt=9346754msec >> >> CLIENT B results: >> # grep agg >> fio128threadsALL/fio128write_ioenginelibaio_iodepth64_direct1_bs512K_20151001015555.out >> WRITE: io=1024.0GB, aggrb=483254KB/s, minb=3775KB/s, maxb=3782KB/s, >> mint=2217808msec, maxt=2221896msec >> >> 4./ If I actually monitor the memory usage of ceph-fuse during the I/O >> tests, I see that >> >> CLIENT A: ceph-fuse does not seem to go behond 7GB of VMEM and 1 GB of RMEM. >> CLIENT B: ceph-fuse uses 11 GB of VMEM and 7 GB of RMEM. >> >> 5./ These results make me think that caching is playing a critical role in >> these results. >> >> My questions are the following: >> >> a./ Why CLIENT B uses more memory than CLIENT A? My hint is that there is a >> network bottleneck between CLIENT B and the Ceph Cluster, and memory is more >> used because of that. > > This is weird, and I don't have an explanation. I would be surprised > if network latency was influencing timing enough to create such a > dramatic difference in caching behaviour. > > Are both clients running the same version of ceph-fuse and the same > version of the kernel? Yeah at 483254KB/s (~480MB/s!) you're clearly exceeding how much the network can actually support and are just writing into RAM. Something in the stack is not actually forcing the data to go out to the OSDs before being acknowledged. Check that all your fio settings are the same and that directIO is actually doing what you expect, that you haven't disabled any of that stuff in the kernel, etc. -Greg > >> b/ Is the FIO write performance better in CLIENT B a consequence of the fact >> that it is using more memory than client A? > > Seems a reasonable inference, but it's still all very weird! > >> c./ Is there a parameters we can set for the CEPHFS clients to limit the >> amount of memory they can use? > > You can limit the caching inside ceph-fuse by setting > client_cache_size (metadata cache entries) and client_oc_size (max > data cache). However, there'll also be some caching inside the kernel > (which you can probably control somehow but I don't know off the top > of my head). > > Cheers, > John > >> >> Cheers >> Goncalo >> >> -- >> Goncalo Borges >> Research Computing >> ARC Centre of Excellence for Particle Physics at the Terascale >> School of Physics A28 | University of Sydney, NSW 2006 >> T: +61 2 93511937 >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |