Thanks for the detailed inquiry. We use HDD with WAL/DB on SSD. The Ceph servers have of lots of RAM and many CPU cores. We are looking for a general purpose approach – running reasonably well in most cases is better than a perfect solution for one use case. This is the existing file size distribution.For the future let's multiply the number of files by 10 (that's more than 100 TB, I know): 1k: 532069 2k: 54458 4k: 36613 8k: 37139 16k: 726302 32k: 286573 64k: 55841 128k: 30510 256k: 37386 512k: 48462 1M: 9461 2M: 4707 4M: 9233 8M: 4816 16M: 3059 32M: 2268 64M: 4314 128M: 17017 256M: 7263 512M: 1917 1G: 1561 2G: 1342 4G: 670 8G: 493 16G: 238 32G: 121 64G: 15 128G: 10 256G: 5 512G: 4 1T: 3 2T: 2 There will be home directories for a few hundred users, and dozens of data dirs with thousands of files between 10-100 kB, which will be processed one by one or in parallel. In this process, some small files are written, but in the main usage, many files of rather small size are read. (A database would be better suited for this, but I have no influence on that.) I would prefer not to create separate pools on SSD, but to use the RAM (some spare servers with 128 GB - 512 GB) for Metadata caching. Thank you for your encouragement! Erich Am So., 15. Jan. 2023 um 16:29 Uhr schrieb Darren Soothill < darren.soothill@xxxxxxxx>: > There are a few details missing to allow people to provide you with advice. > > How many files are you expecting to be in this 100TB of capacity? > This really dictates what you are looking for. It could be full of 4K > files which is a very different proposition to it being full of 100M files. > > What sort of media is this file system made up of? > If you have 10’s millions of files on HDD then you are going to be wanting > a separate metadata pool for CephFS on some much faster storage. > > What is the sort of use case that you are expecting for this storage? > You say it is heavily used but what does that really mean? > You have a 1000 HPC nodes all trying to access millions of 4K files? > Or are you using it as a more general purpose file system for say home > directories? > > > > Darren Soothill > > Looking for help with your Ceph cluster? Contact us at https://croit.io/ > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263 > Web: https://croit.io/ | YouTube: https://goo.gl/PGE1Bx > > > > On 15 Jan 2023, at 09:26, E Taka <0etaka0@xxxxxxxxx> wrote: > > Ceph 17.2.5: > > Hi, > > I'm looking for a reasonable and useful MDS configuration for a – in > future, no experiences until now – heavily used CephFS (~100TB). > For example, does it make a difference to increase the > mds_cache_memory_limit or the number of MDS instances? > > The hardware does not set any limits, I just want to know where the default > values can be optimized usefully before problem occur. > > Thanks, > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |