Re: Stat speed for objects in ceph

[Gregory Farnum <gfarnum@xxxxxxxxxx>]

In librados getting a stat is basically equivalent to reading a small object; there's not an index or anything so FileStore needs to descend its folder hierarchy. If looking at metadata for all the objects in the system efficiently is important you'll want to layer an index in somewhere.

-Greg

On Tuesday, September 20, 2016, Iain Buclaw <ibuclaw@xxxxxxxxx> wrote:

Hi,

As a general observation, the speed of calling stat() on any object in
ceph is relatively slow.  I'm probably getting a rate of about 10K per
second using AIO, and even then it is really *really* bursty, to the
point where there could be 5 seconds of activity going in one
direction, then the callback thread wakes up and processes all queued
completions in a single blast.

At our current rate with more than 1 billion objects in a pool, it's
looking like if I was to check the existence of every object, that it
would take around 19-24 hours to complete.

Granted that our starting point before beginning some migrations to
Ceph was around 1 hour to check the existence of every object, this is
something of a concern.  Are there any ways via librados to improve
the throughput of processing objects?

Adding more instances or sharding work doesn't seem to increase the
overall throughput at all.  And cache won't help either, there is no
determinism in what's accessed, and given the size of the pool OS
filesystem cache is useless anyway.

Thanks,
--
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com