On Thu, Sep 3, 2015 at 11:58 PM, Kyle Hutson <kylehutson@xxxxxxx> wrote: > I was wondering if anybody could give me some insight as to how CephFS does > its caching - read-caching in particular. > > We are using CephFS with an EC pool on the backend with a replicated cache > pool in front of it. We're seeing some very slow read times. Trying to > compute an md5sum on a 15GB file twice in a row (so it should be in cache) > takes the time from 23 minutes down to 17 minutes, but this is over a 10Gbps > network and with a crap-ton of OSDs (over 300), so I would expect it to be > down in the 2-3 minute range. A single sequential read won't necessarily promote an object into the cache pool (although if you're using Hammer I think it will), so you want to check if it's actually getting promoted into the cache before assuming that's happened. > > I'm just trying to figure out what we can do to increase the performance. I > have over 300 TB of live data that I have to be careful with, though, so I > have to have some level of caution. > > Is there some other caching we can do (client-side or server-side) that > might give us a decent performance boost? Which client are you using for this testing? Have you looked at the readahead settings? That's usually the big one; if you're only asking for 4KB at once then stuff is going to be slow no matter what (a single IO takes at minimum about 2 milliseconds right now, although the RADOS team is working to improve that). -Greg > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |