Re: [CephFS] Completely exclude some MDS rank from directory processing

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Eugen,

During the talk you've mentioned, Dan said there's a hard coded limit of 256 MDSs per cluster. So with one active and one standby-ish MDSs per filesystem, that would be 128 filesystems at max per cluster.
Mark said he got 120 but.. things start to get wacky by 80. :-)

More fun to come, for sure.

Cheers,
Frédéric.

[1] https://youtu.be/qiCE1Ifws80?t=2602

----- Le 21 Nov 24, à 9:36, Eugen Block eblock@xxxxxx a écrit :

> I'm not aware of any hard limit for the number of Filesystems, but
> that doesn't really mean very much. IIRC, last week during a Clyso
> talk at Eventbrite I heard someone say that they deployed around 200
> Filesystems or so, I don't remember if it was a production environment
> or just a lab environment. I assume that you would probably be limited
> by the number of OSDs/PGs rather than by the number of Filesystems,
> 200 Filesystems require at least 400 pools. But maybe someone else has
> more experience in scaling CephFS that way. What we did was to scale
> the number of active MDS daemons for one CephFS. I believe in the end
> the customer had 48 MDS daemons on three MDS servers, 16 of them were
> active with directory pinning, at that time they had 16 standby-replay
> and 16 standby daemons. But it turned out that standby-replay didn't
> help their use case, so we disabled standby-replay.
> 
> Can you show the entire 'ceph fs status' output? Any maybe also 'ceph
> fs dump'?
> 
> Zitat von Александр Руденко <a.rudikk@xxxxxxxxx>:
> 
>>>
>>> Just for testing purposes, have you tried pinning rank 1 to some other
>>> directory? Does it still break the CephFS if you stop it?
>>
>>
>> Yes, nothing changed.
>>
>> It's no problem that FS hangs when one of the ranks goes down, we will have
>> standby-reply for all ranks. I don't like that rank which is not pinned to
>> some dir handled some io of this dir or from clients which work with this
>> dir.
>> I mean that I can't robustly and fully separate client IO by ranks.
>>
>> Would it be an option to rather use multiple Filesystems instead of
>>> multi-active for one CephFS?
>>
>>
>> Yes, it's an option. But it is much more complicated in our case. Btw, do
>> you know how many different FS can be created in one cluster? Maybe you
>> know some potential problems with 100-200 FSs in one cluster?
>>
>> ср, 20 нояб. 2024 г. в 17:50, Eugen Block <eblock@xxxxxx>:
>>
>>> Ah, I misunderstood, I thought you wanted an even distribution across
>>> both ranks.
>>> Just for testing purposes, have you tried pinning rank 1 to some other
>>> directory? Does it still break the CephFS if you stop it? I'm not sure
>>> if you can prevent rank 1 from participating, I haven't looked into
>>> all the configs in quite a while. Would it be an option to rather use
>>> multiple Filesystems instead of multi-active for one CephFS?
>>>
>>> Zitat von Александр Руденко <a.rudikk@xxxxxxxxx>:
>>>
>>> > No it's not a typo. It's misleading example)
>>> >
>>> > dir1 and dir2 are pinned to rank 0, but FS and dir1,dir2 can't work
>>> without
>>> > rank 1.
>>> > rank 1 is used for something when I work with this dirs.
>>> >
>>> > ceph 16.2.13, metadata balancer and policy based balancing not used.
>>> >
>>> > ср, 20 нояб. 2024 г. в 16:33, Eugen Block <eblock@xxxxxx>:
>>> >
>>> >> Hi,
>>> >>
>>> >> > After pinning:
>>> >> > setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir1
>>> >> > setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir2
>>> >>
>>> >> is this a typo? If not, you did pin both directories to the same rank.
>>> >>
>>> >> Zitat von Александр Руденко <a.rudikk@xxxxxxxxx>:
>>> >>
>>> >> > Hi,
>>> >> >
>>> >> > I try to distribute all top level dirs in CephFS by different MDS
>>> ranks.
>>> >> > I have two active MDS with rank *0* and *1 *and I have 2 top dirs like
>>> >> > */dir1* and* /dir2*.
>>> >> >
>>> >> > After pinning:
>>> >> > setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir1
>>> >> > setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir2
>>> >> >
>>> >> > I can see next INOS and DNS distribution:
>>> >> > RANK  STATE   MDS     ACTIVITY     DNS    INOS   DIRS   CAPS
>>> >> >  0    active   c   Reqs:    127 /s  12.6k  12.5k   333    505
>>> >> >  1    active   b   Reqs:    11 /s    21     24     19      1
>>> >> >
>>> >> > When I write to dir1 I can see a small amount on Reqs: in rank 1.
>>> >> >
>>> >> > Events in journal of MDS with rank 1:
>>> >> > cephfs-journal-tool --rank=fs1:1 event get list
>>> >> >
>>> >> > 2024-11-20T12:24:42.045056+0300 0xc5c1cb UPDATE:
>>> (scatter_writebehind)
>>> >> >   A2037D53
>>> >> > 2024-11-20T12:24:46.935934+0300 0xc5c629 SESSION:  ()
>>> >> > 2024-11-20T12:24:47.192012+0300 0xc5c7cd UPDATE:  (lock inest
>>> accounted
>>> >> > scatter stat update)
>>> >> > 2024-11-20T12:24:47.904717+0300 0xc5ca0b SESSION:  ()
>>> >> > 2024-11-20T12:26:46.912719+0300 0xc5ca98 SESSION:  ()
>>> >> > 2024-11-20T12:26:47.910806+0300 0xc5cc3c SESSION:  ()
>>> >> > 2024-11-20T12:27:35.746239+0300 0xc5ccc9 SESSION:  ()
>>> >> > 2024-11-20T12:28:46.923812+0300 0xc5ce63 SESSION:  ()
>>> >> > 2024-11-20T12:28:47.903066+0300 0xc5d007 SESSION:  ()
>>> >> > 2024-11-20T12:29:08.063326+0300 0xc5d094 EXPORT:  ()
>>> >> >   di1/A2037D53
>>> >> > 2024-11-20T12:30:46.909621+0300 0xc5d96f SESSION:  ()
>>> >> > 2024-11-20T12:30:47.908050+0300 0xc5db13 SESSION:  ()
>>> >> > 2024-11-20T12:32:46.907649+0300 0xc5dba0 SESSION:  ()
>>> >> > 2024-11-20T12:32:47.905962+0300 0xc5dd44 SESSION:  ()
>>> >> > 2024-11-20T12:34:44.349348+0300 0xc5ddd1 SESSIONS:  ()
>>> >> >
>>> >> > But the main problem, when I stop MDS rank 1 (without any kind of
>>> >> standby)
>>> >> > - FS hangs for all actions.
>>> >> > Is this correct? Is it possible to completely exclude rank 1 from
>>> >> > processing dir1 and not stop io when rank 1 goes down?
>>> >> > _______________________________________________
>>> >> > ceph-users mailing list -- ceph-users@xxxxxxx
>>> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> ceph-users mailing list -- ceph-users@xxxxxxx
>>> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>> >>
>>>
>>>
>>>
>>>
> 
> 
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux