On Wed, Sep 9, 2015 at 4:26 PM, Kyle Hutson <kylehutson@xxxxxxx> wrote: > > > On Wed, Sep 9, 2015 at 9:34 AM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote: >> >> On Wed, Sep 9, 2015 at 3:27 PM, Kyle Hutson <kylehutson@xxxxxxx> wrote: >> > We are using Hammer - latest released version. How do I check if it's >> > getting promoted into the cache? >> >> Umm...that's a good question. You can run rados ls on the cache pool, >> but that's not exactly scalable; you can turn up logging and dig into >> them to see if redirects are happening, or watch the OSD operations >> happening via the admin socket. But I don't know if there's a good >> interface for users to just query the cache state of a single object. >> :/ > > > even using 'rados ls', I (naturally) get cephfs object names - is there a > way to see a filename -> objectname conversion ... or objectname -> filename > ? The object name is <inode number>.<object number in file>. So you can look at the file inode and then see which of its objects are actually in the pool. -Greg > >> >> > We're using the latest ceph kernel client. Where do I poke at readahead >> > settings there? >> >> Just the standard kernel readahead settings; I'm not actually familiar >> with how to configure those but I don't believe Ceph's are in any way >> special. What do you mean by "latest ceph kernel client"; are you >> running one of the developer testing kernels or something? > > > No, just what comes with the latest stock kernel. Sorry for any confusion. > >> >> I think >> Ilya might have mentioned some issues with readahead being >> artificially blocked, but that might have only been with RBD. >> >> Oh, are the files you're using sparse? There was a bug with sparse >> files not filling in pages that just got patched yesterday or >> something. > > > No, these are not sparse files. Just really big. > >> >> > >> > On Tue, Sep 8, 2015 at 8:29 AM, Gregory Farnum <gfarnum@xxxxxxxxxx> >> > wrote: >> >> >> >> On Thu, Sep 3, 2015 at 11:58 PM, Kyle Hutson <kylehutson@xxxxxxx> >> >> wrote: >> >> > I was wondering if anybody could give me some insight as to how >> >> > CephFS >> >> > does >> >> > its caching - read-caching in particular. >> >> > >> >> > We are using CephFS with an EC pool on the backend with a replicated >> >> > cache >> >> > pool in front of it. We're seeing some very slow read times. Trying >> >> > to >> >> > compute an md5sum on a 15GB file twice in a row (so it should be in >> >> > cache) >> >> > takes the time from 23 minutes down to 17 minutes, but this is over a >> >> > 10Gbps >> >> > network and with a crap-ton of OSDs (over 300), so I would expect it >> >> > to >> >> > be >> >> > down in the 2-3 minute range. >> >> >> >> A single sequential read won't necessarily promote an object into the >> >> cache pool (although if you're using Hammer I think it will), so you >> >> want to check if it's actually getting promoted into the cache before >> >> assuming that's happened. >> >> >> >> > >> >> > I'm just trying to figure out what we can do to increase the >> >> > performance. I >> >> > have over 300 TB of live data that I have to be careful with, though, >> >> > so >> >> > I >> >> > have to have some level of caution. >> >> > >> >> > Is there some other caching we can do (client-side or server-side) >> >> > that >> >> > might give us a decent performance boost? >> >> >> >> Which client are you using for this testing? Have you looked at the >> >> readahead settings? That's usually the big one; if you're only asking >> >> for 4KB at once then stuff is going to be slow no matter what (a >> >> single IO takes at minimum about 2 milliseconds right now, although >> >> the RADOS team is working to improve that). >> >> -Greg >> >> >> >> > >> >> > _______________________________________________ >> >> > ceph-users mailing list >> >> > ceph-users@xxxxxxxxxxxxxx >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > >> > >> > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |