Can you run same test several times? Not one, two or three time, just more several times. And check more in detail. For instance descriptor, statics of networking served in /sys/class/net/<interface>/statistics/* and other things. If every single result is same, there would be problems in some layer, networking, fs, vm or whatever. And make sure there is ZERO difference between A and B except your description about the difference between both. The result you show us, actually, should not be. But potentially it could be because of some potential bottleneck. Thank you for your interesting result of your benchmarking anyhow. - Shinobu ----- Original Message ----- From: "John Spray" <jspray@xxxxxxxxxx> To: "Goncalo Borges" <goncalo@xxxxxxxxxxxxxxxxxxx> Cc: "Ceph-User" <ceph-users@xxxxxxxx> Sent: Friday, October 2, 2015 5:57:45 PM Subject: Re: ceph-fuse and its memory usage On Fri, Oct 2, 2015 at 2:42 AM, Goncalo Borges <goncalo@xxxxxxxxxxxxxxxxxxx> wrote: > Dear CephFS Gurus... > > I have a question regarding ceph-fuse and its memory usage. > > 1./ My Ceph and CephFS setups are the following: > > Ceph: > a. ceph 9.0.3 > b. 32 OSDs distributed in 4 servers (8 OSD per server). > c. 'osd pool default size = 3' and 'osd pool default min size = 2' > d. All servers running Centos6.7 > > CephFS: > e. a single mds > f. dedicated pools for data and metadata > g. clients in different locations / sites mounting CephFS via ceph-fuse > h. All servers and clients running Centos6.7 > > 2./ I have been running fio tests in two CephFS clients: > - Client A is in the same data center as all OSDs connected at 1 GbE > - Client B is in a different data center (in another city) also > connected at 1 GbE. However, I've seen that the connection is problematic, > and sometimes, the network performance is well bellow the theoretical 1 Gbps > limit. > - Client A has 24 GB RAM + 98 GB of SWAP and client B has 48 GB of RAM + > 98 GB of SWAP > > 3./ I have been running some fio write tests (with 128 threads) in both > clients, and surprisingly, the results show that the aggregated throughput > is better for client B than client A. > > CLIENT A results: > # grep agg > fio128threadsALL/fio128write_ioenginelibaio_iodepth64_direct1_bs512K_20151001015558.out > WRITE: io=1024.0GB, aggrb=114878KB/s, minb=897KB/s, maxb=1785KB/s, > mint=4697347msec, maxt=9346754msec > > CLIENT B results: > # grep agg > fio128threadsALL/fio128write_ioenginelibaio_iodepth64_direct1_bs512K_20151001015555.out > WRITE: io=1024.0GB, aggrb=483254KB/s, minb=3775KB/s, maxb=3782KB/s, > mint=2217808msec, maxt=2221896msec > > 4./ If I actually monitor the memory usage of ceph-fuse during the I/O > tests, I see that > > CLIENT A: ceph-fuse does not seem to go behond 7GB of VMEM and 1 GB of RMEM. > CLIENT B: ceph-fuse uses 11 GB of VMEM and 7 GB of RMEM. > > 5./ These results make me think that caching is playing a critical role in > these results. > > My questions are the following: > > a./ Why CLIENT B uses more memory than CLIENT A? My hint is that there is a > network bottleneck between CLIENT B and the Ceph Cluster, and memory is more > used because of that. This is weird, and I don't have an explanation. I would be surprised if network latency was influencing timing enough to create such a dramatic difference in caching behaviour. Are both clients running the same version of ceph-fuse and the same version of the kernel? > b/ Is the FIO write performance better in CLIENT B a consequence of the fact > that it is using more memory than client A? Seems a reasonable inference, but it's still all very weird! > c./ Is there a parameters we can set for the CEPHFS clients to limit the > amount of memory they can use? You can limit the caching inside ceph-fuse by setting client_cache_size (metadata cache entries) and client_oc_size (max data cache). However, there'll also be some caching inside the kernel (which you can probably control somehow but I don't know off the top of my head). Cheers, John > > Cheers > Goncalo > > -- > Goncalo Borges > Research Computing > ARC Centre of Excellence for Particle Physics at the Terascale > School of Physics A28 | University of Sydney, NSW 2006 > T: +61 2 93511937 > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |