On Fri, Nov 25, 2016 at 8:16 AM, JiaJia Zhong <zhongjiajia@xxxxxxxxxxxx> wrote:
confusing questions: (ceph0.94)1. Is there any way to cache the whole metadata datas into MDS's memory ? (metadata osds dates-----async----> MDS memory)I dunno if I misunderstand the role of mds :(, so many post threads that advising Using SSD osds for metadata.the metadata stores Inode information for files. Yes, It's fast to stat, ls, readdir for cephfs,but If metadatas could be cached in memory, metadata osds datas -----async----> MDS memory, I guess, this may be better ?we can use ssd journals, so write speed would not be the bottleneck. cached mdatadatas are not large even if there are huge number of files. ( I got that MooseFS strores all metadata in memory ?)
The MDS does cache your metadata in memory, but it also needs to quickly write it safely to disk to fully commit a metadata operation and allow clients to proceed. Even if your metadata fits entirely in memory (i.e. you have fewer than mds_cache_size files) you will still want a fast metadata pool.
John
2.Any descriptions for the Journal under hood ? though it's like swap partition for Linux ~using a Intel PCIE SSD as journals of HDD osds,I ran command blow for a rough benchmark all of the osds simultaneously，# for i in $(ps aux | grep osd | awk '{print $14}' | grep -v "^$" | sort); do ceph tell osd.$i bench & donecompared to anther HOST without SSD journal, these got a better bytes_per_sec, rising about more than 100%.HDD journal OSDS 30MB/s ------> HDD OSDS with SSD journal more than 60MB/s. (12 osds/host, hosts are almost same)MB/s (HDD journal + HDD)
39 35 35 35 33 31 29 26 26 26 26 25 the top 39MB/s one an stata SSD OSD with stata SSD journal, but the speed seems to be not faster than the otheres with HDD Journal + HDD Data.MB/s (PCIE SSD Journal + HDD)
195 129 92 88 71 71 65 61 57 54 52 50 that 195MB/s is PCIE SSD Journal + SSD Data, the speed seems to be very fast. others are PCIE SSD Journal + HDD Data,"bytes_per_sec": 166451390.000000 for single bench on (PCIE Journal + HDD) 158.74MB/s"bytes_per_sec": 78472933.000000 for single bench on (HDD Journal + HDD) 74.83MB/sIt seems that "data ---> HDD Journal" is probable the main bottleneck ? how to track thisdata ----> SSD Journal ------> osd data partitiondata ----> HDD Journal -----> osd data partition3.any cache or memory suggestion for better performance for cephfs?key ceph.conf as below[global]osd pool default size = 2osd pool default min size = 1osd pool default pg num = 512osd pool default pgp num = 512osd journal size = 10000[mds]mds cache size = 11474836[osd]osd op threads = 4filestore op threads = 4osd crush update on start = false#256Mosd max write size = 256#256Mjournal max write bytes = 268435456
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
