Am 27.05.2014 08:37, schrieb Haomai Wang: > I'm not full sure the correctness of changes although it seemed ok to > me. And I apply these changes to product env and no problems. Do you have a branch in your yuyuyu github account for this? > On Tue, May 27, 2014 at 2:05 PM, Stefan Priebe - Profihost AG > <s.priebe@xxxxxxxxxxxx> wrote: >> Am 27.05.2014 06:42, schrieb Haomai Wang: >>> On Tue, May 27, 2014 at 4:29 AM, Stefan Priebe <s.priebe@xxxxxxxxxxxx> wrote: >>>> Hi Haomai, >>>> >>>> regarding the FDCache problems you're seeing. Isn't this branch interesting >>>> for you? Have you ever tested it? >>>> >>>> http://lists.ceph.com/pipermail/ceph-commit-ceph.com/2014-January/007399.html >>>> >>> >>> Yes, I noticed it. But my main job is improving performance on 0.67.5 >>> version. Before this branch, my improvement on this problem is avoid >>> lfn_find in omap* methods with FileStore >>> class.(https://www.mail-archive.com/ceph-devel@xxxxxxxxxxxxxxx/msg18505.html) >> >> Avoids mean just remove them? Are they not needed? Do you have a branch >> for this? >> >>>> Greets, >>>> Stefan >>>> >>>> Am 09.04.2014 12:05, schrieb Haomai Wang: >>>> >>>>> Hi all, >>>>> >>>>> I would like to share some ideas about how to improve performance on >>>>> ceph with SSD. Not much preciseness. >>>>> >>>>> Our ssd is 500GB and each OSD own a SSD(journal is on the same SSD). >>>>> ceph version is 0.67.5(Dumping) >>>>> >>>>> At first, we find three bottleneck on filestore: >>>>> 1. fdcache_lock(changed in Firely release) >>>>> 2. lfn_find in omap_* methods >>>>> 3. DBObjectMap header >>>>> >>>>> According to my understanding and the docs in >>>>> >>>>> ObjectStore.h(https://github.com/ceph/ceph/blob/master/src/os/ObjectStore.h), >>>>> I simply remove lfn_find in omap_* and fdcache_lock. I'm not fully >>>>> sure the correctness of this change, but it works well still now. >>>>> >>>>> DBObjectMap header patch is on the pull request queue and may be >>>>> merged in the next feature merge window. >>>>> >>>>> With things above done, we get much performance improvement in disk >>>>> util and benchmark results(3x-4x). >>>>> >>>>> Next, we find fdcache size become the main bottleneck. For example, if >>>>> hot data range is 100GB, we need 25000(100GB/4MB) fd to cache. If hot >>>>> data range is 1TB, we need 250000(1000GB/4MB) fd to cache. With >>>>> increase "filestore_fd_cache_size", the cost of lookup(FDCache) and >>>>> cache miss is expensive and can't be afford. The implementation of >>>>> FDCache isn't O(1). So we only can get high performance on fdcache hit >>>>> range(maybe 100GB with 10240 fdcache size) and more data exceed the >>>>> size of fdcaceh will be disaster. If you want to cache more fd(102400 >>>>> fdcache size), the implementation of FDCache will bring on extra CPU >>>>> cost(can't be ignore) for each op. >>>>> >>>>> Because of the capacity of SSD(several hundreds GB), we try to >>>>> increase the size of rbd object(16MB) so less fd cache is needed. As >>>>> for FDCache implementation, we simply discard SimpleLRU but introduce >>>>> RandomCache. Now we can set much larger fdcache size(near cache all >>>>> fd) with little overload. >>>>> >>>>> With these, we achieve 3x-4x performance improvements on filestore with >>>>> SSD. >>>>> >>>>> Maybe it exists something I missed or something wrong, hope can >>>>> correct me. I hope it can help to improve FileStore on SSD and push >>>>> into master branch. >>>>> >>>> >>> >>> >>> > > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
