Re: Useful MDS configuration for heavily used Cephfs

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Thanks for the detailed inquiry. We use HDD with WAL/DB on SSD. The Ceph
servers have of lots of RAM and many CPU cores. We are looking for
a general purpose approach – running reasonably well in most cases is
better than a perfect solution for one use case.

This is the existing file size distribution.For the future let's multiply
the number of files by 10 (that's more than 100 TB, I know):

  1k: 532069
 2k:  54458
 4k:  36613
 8k:  37139
16k: 726302
32k: 286573
64k:  55841
128k:  30510
256k:  37386
512k:  48462
 1M:   9461
 2M:   4707
 4M:   9233
 8M:   4816
16M:   3059
32M:   2268
64M:   4314
128M:  17017
256M:   7263
512M:   1917
 1G:   1561
 2G:   1342
 4G:    670
 8G:    493
16G:    238
32G:    121
64G:     15
128G:     10
256G:      5
512G:      4
 1T:      3
 2T:      2

There will be home directories for a few hundred users, and dozens of data
dirs with thousands of files between 10-100 kB, which will be processed one
by one or in parallel. In this process, some small files are written, but
in the main usage, many files of rather small size are read.
(A database would be better suited for this, but I have no influence on
that.) I would prefer not to create separate pools on SSD, but to use the
RAM (some spare servers with 128 GB - 512 GB) for Metadata caching.

Thank you for your encouragement!
Erich

Am So., 15. Jan. 2023 um 16:29 Uhr schrieb Darren Soothill <
darren.soothill@xxxxxxxx>:

> There are a few details missing to allow people to provide you with advice.
>
> How many files are you expecting to be in this 100TB of capacity?
> This really dictates what you are looking for. It could be full of 4K
> files which is a very different proposition to it being full of 100M files.
>
> What sort of media is this file system made up of?
> If you have 10’s millions of files on HDD then you are going to be wanting
> a separate metadata pool for CephFS on some much faster storage.
>
> What is the sort of use case that you are expecting for this storage?
> You say it is heavily used but what does that really mean?
> You have a 1000 HPC nodes all trying to access millions of 4K files?
> Or are you using it as a more general purpose file system for say home
> directories?
>
>
>
> Darren Soothill
>
> Looking for help with your Ceph cluster? Contact us at https://croit.io/
>
> croit GmbH, Freseniusstr. 31h, 81247 Munich
> CEO: Martin Verges - VAT-ID: DE310638492
> Com. register: Amtsgericht Munich HRB 231263
> Web: https://croit.io/ | YouTube: https://goo.gl/PGE1Bx
>
>
>
> On 15 Jan 2023, at 09:26, E Taka <0etaka0@xxxxxxxxx> wrote:
>
> Ceph 17.2.5:
>
> Hi,
>
> I'm looking for a reasonable and useful MDS configuration for a – in
> future, no experiences until now – heavily used CephFS (~100TB).
> For example, does it make a difference to increase the
> mds_cache_memory_limit or the number of MDS instances?
>
> The hardware does not set any limits, I just want to know where the default
> values can be optimized usefully before problem occur.
>
> Thanks,
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux