Ok, great. Some numbers for you: I have a filesystem of 50 million files, 5.4 TB. The data pool is on HDD OSDs with Optane DB/WAL, size=3. The metadata pool (Optane OSDs) has 17GiB "stored", 20GiB "used", at size=3. 5.18M objects. When doing parallel rsyncs, with ~14M inodes open, the MDS cache goes to about 40GiB but it remains stable. MDS CPU usage goes to about 400% (4 cores worth, spread across 6-8 processes). Hope you find this useful. On Fri, Jul 26, 2019 at 11:05 AM Stefan Kooman <stefan@xxxxxx> wrote: > > Quoting Nathan Fish (lordcirth@xxxxxxxxx): > > MDS CPU load is proportional to metadata ops/second. MDS RAM cache is > > proportional to # of files (including directories) in the working set. > > Metadata pool size is proportional to total # of files, plus > > everything in the RAM cache. I have seen that the metadata pool can > > balloon 8x between being idle, and having every inode open by a > > client. > > The main thing I'd recommend is getting SSD OSDs to dedicate to the > > metadata pools, and SSDs for the HDD OSD's DB/WAL. NVMe if you can. If > > you put that much metadata on only HDDs, it's going to be slow. > > Only SSD for OSD data pool and NVMe for metadata pool, so that should be > fine. Besides the initial loading of that many files / directories this > workload shouldn't be any problem. > > Thanks for your feedback. > > Gr. Stefan > > -- > | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 > | GPG: 0xD14839C6 +31 318 648 688 / info@xxxxxx _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |