Hi ceph-users,
We are using Ceph (radosgw) to store user generated images, as GET latency is critical for us, most recently I did some investigation over the GET path to understand where time spend.
I first confirmed that the latency came from OSD (read op), so that we instrumented code to trace the GET request (read op at OSD side, to be more specific, each object with size [512K + 4M * x] are splitted into [1 + x] chunks, each chunk needs one read op ), for each read op, it needs to go through the following steps:
1. Dispatch and take by a op thread to process (process not started).
0 – 20 ms, 94%
20 – 50 ms, 2%
50 – 100 ms, 2%
100ms+ , 2%
For those having 20ms+ latency, half of them are due to waiting for pg lock (https://github.com/ceph/ceph/blob/dumpling/src/osd/OSD.cc#L7089), another half are yet to be investigated.
2. Get file xattr (‘-‘), which open the file and populate fd cache (https://github.com/ceph/ceph/blob/dumpling/src/os/FileStore.cc#L230).
0 – 20 ms, 80%
20 – 50 ms, 8%
50 – 100 ms, 7%
100ms+ , 5%
The latency either comes from (from more to less): file path lookup (https://github.com/ceph/ceph/blob/dumpling/src/os/HashIndex.cc#L294), file open, or fd cache lookup /add.
Currently objects are store in level 6 or level 7 folder (due to http://tracker.ceph.com/issues/7207, I stopped folder splitting).
3. Get more xattrs, this is fast due to previous fd cache (rarely > 1ms).
4. Read the data.
0 – 20 ms, 84%
20 – 50 ms, 10%
50 – 100 ms, 4%
100ms+ , 2%
I decreased vfs_cache_pressure from its default value 100 to 5 to make VFS favor dentry/inode cache over page cache, unfortunately it does not help.
Long story short, most of the long latency read op comes from file system call (for cold data), as our workload mainly stores objects less than 500KB, so that it generates a large bunch of objects.
I would like to ask if people experienced similar issue and if there is any suggestion I can try to boost the GET performance. On the other hand, PUT could be sacrificed.
Thanks,
Guang
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
