| Thanks Yehuda.
Try looking at the perfcounters, see if there's any other throttling
happening. Also, make sure you have enough pgs for your data pool. One
other thing to try is disabling leveldb xattrs and see if it affects
your latency.
1. There is not throttling happening. 2. According to ceph recommendation, there is 100PGs per OSD and we bump that to 200. 3. Can you elaborate on the xattrs stuff, how could that affect the get latency?
Thanks, Guang On Tue, Feb 18, 2014 at 7:24 AM, Guang Yang < yguang11@xxxxxxxxx> wrote: Hi ceph-users,
We are using Ceph (radosgw) to store user generated images, as GET latency
is critical for us, most recently I did some investigation over the GET path
to understand where time spend.
I first confirmed that the latency came from OSD (read op), so that we
instrumented code to trace the GET request (read op at OSD side, to be more
specific, each object with size [512K + 4M * x] are splitted into [1 + x]
chunks, each chunk needs one read op ), for each read op, it needs to go
through the following steps:
1. Dispatch and take by a op thread to process (process not started).
0 - 20 ms, 94%
20 - 50 ms, 2%
50 - 100 ms, 2%
100ms+ , 2%
For those having 20ms+ latency, half of them are due to waiting for
pg lock (https://github.com/ceph/ceph/blob/dumpling/src/osd/OSD.cc#L7089),
another half are yet to be investigated.
2. Get file xattr ('-'), which open the file and populate fd cache
(https://github.com/ceph/ceph/blob/dumpling/src/os/FileStore.cc#L230).
0 - 20 ms, 80%
20 - 50 ms, 8%
50 - 100 ms, 7%
100ms+ , 5%
The latency either comes from (from more to less): file path lookup
(https://github.com/ceph/ceph/blob/dumpling/src/os/HashIndex.cc#L294), file
open, or fd cache lookup /add.
Currently objects are store in level 6 or level 7 folder (due to
http://tracker.ceph.com/issues/7207, I stopped folder splitting).
3. Get more xattrs, this is fast due to previous fd cache (rarely >
1ms).
4. Read the data.
0 - 20 ms, 84%
20 - 50 ms, 10%
50 - 100 ms, 4%
100ms+ , 2%
I decreased vfs_cache_pressure from its default value 100 to 5 to make VFS
favor dentry/inode cache over page cache, unfortunately it does not help.
Long story short, most of the long latency read op comes from file system
call (for cold data), as our workload mainly stores objects less than 500KB,
so that it generates a large bunch of objects.
I would like to ask if people experienced similar issue and if there is any
suggestion I can try to boost the GET performance. On the other hand, PUT
could be sacrificed.
Try looking at the perfcounters, see if there's any other throttling happening. Also, make sure you have enough pgs for your data pool. One other thing to try is disabling leveldb xattrs and see if it affects your latency. Yehuda
|
