Hi,
we are running CephFS with about 70TB data, > 5 million files and about
100 clients. The MDS is currently colocated on a storage box with 14 OSD
(12 HDD, 2SSD). The box has two E52680v3 CPUs and 128 GB RAM. CephFS
runs fine, but it feels like the metadata operations may need more speed.
Excerpt of MDS perf dump:
"mds": {
"request": 73389282,
"reply": 73389282,
"reply_latency": {
"avgcount": 73389282,
"sum": 259696.749971457
},
"forward": 0,
"dir_fetch": 4094842,
"dir_commit": 720085,
"dir_split": 0,
"inode_max": 5000000,
"inodes": 5000065,
"inodes_top": 320979,
"inodes_bottom": 530518,
"inodes_pin_tail": 4148568,
"inodes_pinned": 4469666,
"inodes_expired": 60001276,
"inodes_with_caps": 4468714,
"caps": 4850520,
"subtrees": 2,
"traverse": 92378836,
"traverse_hit": 75743822,
"traverse_forward": 0,
"traverse_discover": 0,
"traverse_dir_fetch": 1719440,
"traverse_remote_ino": 33,
"traverse_lock": 3952,
"load_cent": 7339063064,
"q": 0,
"exported": 0,
"exported_inodes": 0,
"imported": 0,
"imported_inodes": 0
},....
The setup is expected grow, with regards to the amount of stored data
and the number of clients. The MDS process currently consumes about 36
TB RAM, with 22 TB resident. Since a large part of the MDS run single
threaded, a CPU with less core and more CPU frequency might be a better
choice in this setup.
How well does the MDS performance scale with CPU frequency (given other
latency pathes like network I/O don't matter)? Given the amount of
memory used, does the MDS benefit from larger CPU caches (e.g. E5-2XXX
class cpu), or a smaller cache in faster CPUs a better choice (e.g.
E5-1XXX or E3-1XXXv5)?
Regards,
Burkhard
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
