A 'rados -p cachepool ls' takes about 3 hours - not exactly useful.
I'm intrigued that you say a single read may not promote it into the cache. My understanding is that if you have an EC-backed pool the clients can't talk to them directly, which means they would necessarily be promoted to the cache pool so the client could read it. Is my understanding wrong?
I'm also wondering if it's possible to use RAM as a read-cache layer. Obviously, we don't want this for write-cache because of power outages, motherboard failures, etc., but it seems to make sense for a read-cache. Is that something that's being done, can be done, is going to be done, or has even been considered?
On Wed, Sep 9, 2015 at 10:33 AM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
On Wed, Sep 9, 2015 at 4:26 PM, Kyle Hutson <kylehutson@xxxxxxx> wrote:
>
>
> On Wed, Sep 9, 2015 at 9:34 AM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
>>
>> On Wed, Sep 9, 2015 at 3:27 PM, Kyle Hutson <kylehutson@xxxxxxx> wrote:
>> > We are using Hammer - latest released version. How do I check if it's
>> > getting promoted into the cache?
>>
>> Umm...that's a good question. You can run rados ls on the cache pool,
>> but that's not exactly scalable; you can turn up logging and dig into
>> them to see if redirects are happening, or watch the OSD operations
>> happening via the admin socket. But I don't know if there's a good
>> interface for users to just query the cache state of a single object.
>> :/
>
>
> even using 'rados ls', I (naturally) get cephfs object names - is there a
> way to see a filename -> objectname conversion ... or objectname -> filename
> ?
The object name is <inode number>.<object number in file>. So you can
look at the file inode and then see which of its objects are actually
in the pool.
-Greg
>
>>
>> > We're using the latest ceph kernel client. Where do I poke at readahead
>> > settings there?
>>
>> Just the standard kernel readahead settings; I'm not actually familiar
>> with how to configure those but I don't believe Ceph's are in any way
>> special. What do you mean by "latest ceph kernel client"; are you
>> running one of the developer testing kernels or something?
>
>
> No, just what comes with the latest stock kernel. Sorry for any confusion.
>
>>
>> I think
>> Ilya might have mentioned some issues with readahead being
>> artificially blocked, but that might have only been with RBD.
>>
>> Oh, are the files you're using sparse? There was a bug with sparse
>> files not filling in pages that just got patched yesterday or
>> something.
>
>
> No, these are not sparse files. Just really big.
>
>>
>> >
>> > On Tue, Sep 8, 2015 at 8:29 AM, Gregory Farnum <gfarnum@xxxxxxxxxx>
>> > wrote:
>> >>
>> >> On Thu, Sep 3, 2015 at 11:58 PM, Kyle Hutson <kylehutson@xxxxxxx>
>> >> wrote:
>> >> > I was wondering if anybody could give me some insight as to how
>> >> > CephFS
>> >> > does
>> >> > its caching - read-caching in particular.
>> >> >
>> >> > We are using CephFS with an EC pool on the backend with a replicated
>> >> > cache
>> >> > pool in front of it. We're seeing some very slow read times. Trying
>> >> > to
>> >> > compute an md5sum on a 15GB file twice in a row (so it should be in
>> >> > cache)
>> >> > takes the time from 23 minutes down to 17 minutes, but this is over a
>> >> > 10Gbps
>> >> > network and with a crap-ton of OSDs (over 300), so I would expect it
>> >> > to
>> >> > be
>> >> > down in the 2-3 minute range.
>> >>
>> >> A single sequential read won't necessarily promote an object into the
>> >> cache pool (although if you're using Hammer I think it will), so you
>> >> want to check if it's actually getting promoted into the cache before
>> >> assuming that's happened.
>> >>
>> >> >
>> >> > I'm just trying to figure out what we can do to increase the
>> >> > performance. I
>> >> > have over 300 TB of live data that I have to be careful with, though,
>> >> > so
>> >> > I
>> >> > have to have some level of caution.
>> >> >
>> >> > Is there some other caching we can do (client-side or server-side)
>> >> > that
>> >> > might give us a decent performance boost?
>> >>
>> >> Which client are you using for this testing? Have you looked at the
>> >> readahead settings? That's usually the big one; if you're only asking
>> >> for 4KB at once then stuff is going to be slow no matter what (a
>> >> single IO takes at minimum about 2 milliseconds right now, although
>> >> the RADOS team is working to improve that).
>> >> -Greg
>> >>
>> >> >
>> >> > _______________________________________________
>> >> > ceph-users mailing list
>> >> > ceph-users@xxxxxxxxxxxxxx
>> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >> >
>> >
>> >
>
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
