Still not, I will try to push to master branch On Tue, May 27, 2014 at 2:45 PM, Stefan Priebe - Profihost AG <s.priebe@xxxxxxxxxxxx> wrote: > Am 27.05.2014 08:37, schrieb Haomai Wang: >> I'm not full sure the correctness of changes although it seemed ok to >> me. And I apply these changes to product env and no problems. > > Do you have a branch in your yuyuyu github account for this? > >> On Tue, May 27, 2014 at 2:05 PM, Stefan Priebe - Profihost AG >> <s.priebe@xxxxxxxxxxxx> wrote: >>> Am 27.05.2014 06:42, schrieb Haomai Wang: >>>> On Tue, May 27, 2014 at 4:29 AM, Stefan Priebe <s.priebe@xxxxxxxxxxxx> wrote: >>>>> Hi Haomai, >>>>> >>>>> regarding the FDCache problems you're seeing. Isn't this branch interesting >>>>> for you? Have you ever tested it? >>>>> >>>>> http://lists.ceph.com/pipermail/ceph-commit-ceph.com/2014-January/007399.html >>>>> >>>> >>>> Yes, I noticed it. But my main job is improving performance on 0.67.5 >>>> version. Before this branch, my improvement on this problem is avoid >>>> lfn_find in omap* methods with FileStore >>>> class.(https://www.mail-archive.com/ceph-devel@xxxxxxxxxxxxxxx/msg18505.html) >>> >>> Avoids mean just remove them? Are they not needed? Do you have a branch >>> for this? >>> >>>>> Greets, >>>>> Stefan >>>>> >>>>> Am 09.04.2014 12:05, schrieb Haomai Wang: >>>>> >>>>>> Hi all, >>>>>> >>>>>> I would like to share some ideas about how to improve performance on >>>>>> ceph with SSD. Not much preciseness. >>>>>> >>>>>> Our ssd is 500GB and each OSD own a SSD(journal is on the same SSD). >>>>>> ceph version is 0.67.5(Dumping) >>>>>> >>>>>> At first, we find three bottleneck on filestore: >>>>>> 1. fdcache_lock(changed in Firely release) >>>>>> 2. lfn_find in omap_* methods >>>>>> 3. DBObjectMap header >>>>>> >>>>>> According to my understanding and the docs in >>>>>> >>>>>> ObjectStore.h(https://github.com/ceph/ceph/blob/master/src/os/ObjectStore.h), >>>>>> I simply remove lfn_find in omap_* and fdcache_lock. I'm not fully >>>>>> sure the correctness of this change, but it works well still now. >>>>>> >>>>>> DBObjectMap header patch is on the pull request queue and may be >>>>>> merged in the next feature merge window. >>>>>> >>>>>> With things above done, we get much performance improvement in disk >>>>>> util and benchmark results(3x-4x). >>>>>> >>>>>> Next, we find fdcache size become the main bottleneck. For example, if >>>>>> hot data range is 100GB, we need 25000(100GB/4MB) fd to cache. If hot >>>>>> data range is 1TB, we need 250000(1000GB/4MB) fd to cache. With >>>>>> increase "filestore_fd_cache_size", the cost of lookup(FDCache) and >>>>>> cache miss is expensive and can't be afford. The implementation of >>>>>> FDCache isn't O(1). So we only can get high performance on fdcache hit >>>>>> range(maybe 100GB with 10240 fdcache size) and more data exceed the >>>>>> size of fdcaceh will be disaster. If you want to cache more fd(102400 >>>>>> fdcache size), the implementation of FDCache will bring on extra CPU >>>>>> cost(can't be ignore) for each op. >>>>>> >>>>>> Because of the capacity of SSD(several hundreds GB), we try to >>>>>> increase the size of rbd object(16MB) so less fd cache is needed. As >>>>>> for FDCache implementation, we simply discard SimpleLRU but introduce >>>>>> RandomCache. Now we can set much larger fdcache size(near cache all >>>>>> fd) with little overload. >>>>>> >>>>>> With these, we achieve 3x-4x performance improvements on filestore with >>>>>> SSD. >>>>>> >>>>>> Maybe it exists something I missed or something wrong, hope can >>>>>> correct me. I hope it can help to improve FileStore on SSD and push >>>>>> into master branch. >>>>>> >>>>> >>>> >>>> >>>> >> >> >> -- Best Regards, Wheat -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
