On Wed, Sep 9, 2015 at 3:27 PM, Kyle Hutson <kylehutson@xxxxxxx> wrote: > We are using Hammer - latest released version. How do I check if it's > getting promoted into the cache? Umm...that's a good question. You can run rados ls on the cache pool, but that's not exactly scalable; you can turn up logging and dig into them to see if redirects are happening, or watch the OSD operations happening via the admin socket. But I don't know if there's a good interface for users to just query the cache state of a single object. :/ > > We're using the latest ceph kernel client. Where do I poke at readahead > settings there? Just the standard kernel readahead settings; I'm not actually familiar with how to configure those but I don't believe Ceph's are in any way special. What do you mean by "latest ceph kernel client"; are you running one of the developer testing kernels or something? I think Ilya might have mentioned some issues with readahead being artificially blocked, but that might have only been with RBD. Oh, are the files you're using sparse? There was a bug with sparse files not filling in pages that just got patched yesterday or something. -Greg > > On Tue, Sep 8, 2015 at 8:29 AM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote: >> >> On Thu, Sep 3, 2015 at 11:58 PM, Kyle Hutson <kylehutson@xxxxxxx> wrote: >> > I was wondering if anybody could give me some insight as to how CephFS >> > does >> > its caching - read-caching in particular. >> > >> > We are using CephFS with an EC pool on the backend with a replicated >> > cache >> > pool in front of it. We're seeing some very slow read times. Trying to >> > compute an md5sum on a 15GB file twice in a row (so it should be in >> > cache) >> > takes the time from 23 minutes down to 17 minutes, but this is over a >> > 10Gbps >> > network and with a crap-ton of OSDs (over 300), so I would expect it to >> > be >> > down in the 2-3 minute range. >> >> A single sequential read won't necessarily promote an object into the >> cache pool (although if you're using Hammer I think it will), so you >> want to check if it's actually getting promoted into the cache before >> assuming that's happened. >> >> > >> > I'm just trying to figure out what we can do to increase the >> > performance. I >> > have over 300 TB of live data that I have to be careful with, though, so >> > I >> > have to have some level of caution. >> > >> > Is there some other caching we can do (client-side or server-side) that >> > might give us a decent performance boost? >> >> Which client are you using for this testing? Have you looked at the >> readahead settings? That's usually the big one; if you're only asking >> for 4KB at once then stuff is going to be slow no matter what (a >> single IO takes at minimum about 2 milliseconds right now, although >> the RADOS team is working to improve that). >> -Greg >> >> > >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users@xxxxxxxxxxxxxx >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |