I'm not full sure the correctness of changes although it seemed ok to me. And I apply these changes to product env and no problems. On Tue, May 27, 2014 at 2:05 PM, Stefan Priebe - Profihost AG <s.priebe@xxxxxxxxxxxx> wrote: > Am 27.05.2014 06:42, schrieb Haomai Wang: >> On Tue, May 27, 2014 at 4:29 AM, Stefan Priebe <s.priebe@xxxxxxxxxxxx> wrote: >>> Hi Haomai, >>> >>> regarding the FDCache problems you're seeing. Isn't this branch interesting >>> for you? Have you ever tested it? >>> >>> http://lists.ceph.com/pipermail/ceph-commit-ceph.com/2014-January/007399.html >>> >> >> Yes, I noticed it. But my main job is improving performance on 0.67.5 >> version. Before this branch, my improvement on this problem is avoid >> lfn_find in omap* methods with FileStore >> class.(https://www.mail-archive.com/ceph-devel@xxxxxxxxxxxxxxx/msg18505.html) > > Avoids mean just remove them? Are they not needed? Do you have a branch > for this? > >>> Greets, >>> Stefan >>> >>> Am 09.04.2014 12:05, schrieb Haomai Wang: >>> >>>> Hi all, >>>> >>>> I would like to share some ideas about how to improve performance on >>>> ceph with SSD. Not much preciseness. >>>> >>>> Our ssd is 500GB and each OSD own a SSD(journal is on the same SSD). >>>> ceph version is 0.67.5(Dumping) >>>> >>>> At first, we find three bottleneck on filestore: >>>> 1. fdcache_lock(changed in Firely release) >>>> 2. lfn_find in omap_* methods >>>> 3. DBObjectMap header >>>> >>>> According to my understanding and the docs in >>>> >>>> ObjectStore.h(https://github.com/ceph/ceph/blob/master/src/os/ObjectStore.h), >>>> I simply remove lfn_find in omap_* and fdcache_lock. I'm not fully >>>> sure the correctness of this change, but it works well still now. >>>> >>>> DBObjectMap header patch is on the pull request queue and may be >>>> merged in the next feature merge window. >>>> >>>> With things above done, we get much performance improvement in disk >>>> util and benchmark results(3x-4x). >>>> >>>> Next, we find fdcache size become the main bottleneck. For example, if >>>> hot data range is 100GB, we need 25000(100GB/4MB) fd to cache. If hot >>>> data range is 1TB, we need 250000(1000GB/4MB) fd to cache. With >>>> increase "filestore_fd_cache_size", the cost of lookup(FDCache) and >>>> cache miss is expensive and can't be afford. The implementation of >>>> FDCache isn't O(1). So we only can get high performance on fdcache hit >>>> range(maybe 100GB with 10240 fdcache size) and more data exceed the >>>> size of fdcaceh will be disaster. If you want to cache more fd(102400 >>>> fdcache size), the implementation of FDCache will bring on extra CPU >>>> cost(can't be ignore) for each op. >>>> >>>> Because of the capacity of SSD(several hundreds GB), we try to >>>> increase the size of rbd object(16MB) so less fd cache is needed. As >>>> for FDCache implementation, we simply discard SimpleLRU but introduce >>>> RandomCache. Now we can set much larger fdcache size(near cache all >>>> fd) with little overload. >>>> >>>> With these, we achieve 3x-4x performance improvements on filestore with >>>> SSD. >>>> >>>> Maybe it exists something I missed or something wrong, hope can >>>> correct me. I hope it can help to improve FileStore on SSD and push >>>> into master branch. >>>> >>> >> >> >> -- Best Regards, Wheat -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
