On Tue, Feb 18, 2014 at 7:24 AM, Guang Yang <yguang11@xxxxxxxxx> wrote: > Hi ceph-users, > We are using Ceph (radosgw) to store user generated images, as GET latency > is critical for us, most recently I did some investigation over the GET path > to understand where time spend. > > I first confirmed that the latency came from OSD (read op), so that we > instrumented code to trace the GET request (read op at OSD side, to be more > specific, each object with size [512K + 4M * x] are splitted into [1 + x] > chunks, each chunk needs one read op ), for each read op, it needs to go > through the following steps: > 1. Dispatch and take by a op thread to process (process not started). > 0 - 20 ms, 94% > 20 - 50 ms, 2% > 50 - 100 ms, 2% > 100ms+ , 2% > For those having 20ms+ latency, half of them are due to waiting for > pg lock (https://github.com/ceph/ceph/blob/dumpling/src/osd/OSD.cc#L7089), > another half are yet to be investigated. > > 2. Get file xattr ('-'), which open the file and populate fd cache > (https://github.com/ceph/ceph/blob/dumpling/src/os/FileStore.cc#L230). > 0 - 20 ms, 80% > 20 - 50 ms, 8% > 50 - 100 ms, 7% > 100ms+ , 5% > The latency either comes from (from more to less): file path lookup > (https://github.com/ceph/ceph/blob/dumpling/src/os/HashIndex.cc#L294), file > open, or fd cache lookup /add. > Currently objects are store in level 6 or level 7 folder (due to > http://tracker.ceph.com/issues/7207, I stopped folder splitting). > > 3. Get more xattrs, this is fast due to previous fd cache (rarely > > 1ms). > > 4. Read the data. > 0 - 20 ms, 84% > 20 - 50 ms, 10% > 50 - 100 ms, 4% > 100ms+ , 2% > > I decreased vfs_cache_pressure from its default value 100 to 5 to make VFS > favor dentry/inode cache over page cache, unfortunately it does not help. > > Long story short, most of the long latency read op comes from file system > call (for cold data), as our workload mainly stores objects less than 500KB, > so that it generates a large bunch of objects. > > I would like to ask if people experienced similar issue and if there is any > suggestion I can try to boost the GET performance. On the other hand, PUT > could be sacrificed. > Try looking at the perfcounters, see if there's any other throttling happening. Also, make sure you have enough pgs for your data pool. One other thing to try is disabling leveldb xattrs and see if it affects your latency. Yehuda _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |