>
> Can you show the entire 'ceph fs status' output? Any maybe also 'ceph
> fs dump'?
Nothing special, just smoll test cluster.
fs1 - 10 clients
===
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active a Reqs: 0 /s 18.7k 18.4k 351 513
1 active b Reqs: 0 /s 21 24 16 1
POOL TYPE USED AVAIL
fs1_meta metadata 116M 3184G
fs1_data data 23.8G 3184G
STANDBY MDS
c
fs dump
e48
enable_multiple, ever_enabled_multiple: 1,1
default compat: compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'fs1' (1)
fs_name fs1
epoch 47
flags 12
created 2024-10-15T18:55:10.905035+0300
modified 2024-11-21T10:55:12.688598+0300
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 943
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds
uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline
data,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=12200812,1=11974933}
failed
damaged
stopped
data_pools [7]
metadata_pool 6
inline_data disabled
balancer
standby_count_wanted 1
[mds.a{0:12200812} state up:active seq 13 addr [v2:
10.7.1.115:6842/1955635987,v1:10.7.1.115:6843/1955635987] compat
{c=[1],r=[1],i=[7ff]}]
[mds.b{1:11974933} state up:active seq 5 addr [v2:
10.7.1.116:6840/536741454,v1:10.7.1.116:6841/536741454] compat
{c=[1],r=[1],i=[7ff]}]
Standby daemons:
[mds.c{-1:11704322} state up:standby seq 1 addr [v2:
10.7.1.117:6848/84247504,v1:10.7.1.117:6849/84247504] compat
{c=[1],r=[1],i=[7ff]}]
чт, 21 нояб. 2024 г. в 11:36, Eugen Block <eblock@xxxxxx>:
> I'm not aware of any hard limit for the number of Filesystems, but
> that doesn't really mean very much. IIRC, last week during a Clyso
> talk at Eventbrite I heard someone say that they deployed around 200
> Filesystems or so, I don't remember if it was a production environment
> or just a lab environment. I assume that you would probably be limited
> by the number of OSDs/PGs rather than by the number of Filesystems,
> 200 Filesystems require at least 400 pools. But maybe someone else has
> more experience in scaling CephFS that way. What we did was to scale
> the number of active MDS daemons for one CephFS. I believe in the end
> the customer had 48 MDS daemons on three MDS servers, 16 of them were
> active with directory pinning, at that time they had 16 standby-replay
> and 16 standby daemons. But it turned out that standby-replay didn't
> help their use case, so we disabled standby-replay.
>
> Can you show the entire 'ceph fs status' output? Any maybe also 'ceph
> fs dump'?
>
> Zitat von Александр Руденко <a.rudikk@xxxxxxxxx>:
>
> >>
> >> Just for testing purposes, have you tried pinning rank 1 to some other
> >> directory? Does it still break the CephFS if you stop it?
> >
> >
> > Yes, nothing changed.
> >
> > It's no problem that FS hangs when one of the ranks goes down, we will
> have
> > standby-reply for all ranks. I don't like that rank which is not pinned
> to
> > some dir handled some io of this dir or from clients which work with this
> > dir.
> > I mean that I can't robustly and fully separate client IO by ranks.
> >
> > Would it be an option to rather use multiple Filesystems instead of
> >> multi-active for one CephFS?
> >
> >
> > Yes, it's an option. But it is much more complicated in our case. Btw, do
> > you know how many different FS can be created in one cluster? Maybe you
> > know some potential problems with 100-200 FSs in one cluster?
> >
> > ср, 20 нояб. 2024 г. в 17:50, Eugen Block <eblock@xxxxxx>:
> >
> >> Ah, I misunderstood, I thought you wanted an even distribution across
> >> both ranks.
> >> Just for testing purposes, have you tried pinning rank 1 to some other
> >> directory? Does it still break the CephFS if you stop it? I'm not sure
> >> if you can prevent rank 1 from participating, I haven't looked into
> >> all the configs in quite a while. Would it be an option to rather use
> >> multiple Filesystems instead of multi-active for one CephFS?
> >>
> >> Zitat von Александр Руденко <a.rudikk@xxxxxxxxx>:
> >>
> >> > No it's not a typo. It's misleading example)
> >> >
> >> > dir1 and dir2 are pinned to rank 0, but FS and dir1,dir2 can't work
> >> without
> >> > rank 1.
> >> > rank 1 is used for something when I work with this dirs.
> >> >
> >> > ceph 16.2.13, metadata balancer and policy based balancing not used.
> >> >
> >> > ср, 20 нояб. 2024 г. в 16:33, Eugen Block <eblock@xxxxxx>:
> >> >
> >> >> Hi,
> >> >>
> >> >> > After pinning:
> >> >> > setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir1
> >> >> > setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir2
> >> >>
> >> >> is this a typo? If not, you did pin both directories to the same
> rank.
> >> >>
> >> >> Zitat von Александр Руденко <a.rudikk@xxxxxxxxx>:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > I try to distribute all top level dirs in CephFS by different MDS
> >> ranks.
> >> >> > I have two active MDS with rank *0* and *1 *and I have 2 top dirs
> like
> >> >> > */dir1* and* /dir2*.
> >> >> >
> >> >> > After pinning:
> >> >> > setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir1
> >> >> > setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir2
> >> >> >
> >> >> > I can see next INOS and DNS distribution:
> >> >> > RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
> >> >> > 0 active c Reqs: 127 /s 12.6k 12.5k 333 505
> >> >> > 1 active b Reqs: 11 /s 21 24 19 1
> >> >> >
> >> >> > When I write to dir1 I can see a small amount on Reqs: in rank 1.
> >> >> >
> >> >> > Events in journal of MDS with rank 1:
> >> >> > cephfs-journal-tool --rank=fs1:1 event get list
> >> >> >
> >> >> > 2024-11-20T12:24:42.045056+0300 0xc5c1cb UPDATE:
> >> (scatter_writebehind)
> >> >> > A2037D53
> >> >> > 2024-11-20T12:24:46.935934+0300 0xc5c629 SESSION: ()
> >> >> > 2024-11-20T12:24:47.192012+0300 0xc5c7cd UPDATE: (lock inest
> >> accounted
> >> >> > scatter stat update)
> >> >> > 2024-11-20T12:24:47.904717+0300 0xc5ca0b SESSION: ()
> >> >> > 2024-11-20T12:26:46.912719+0300 0xc5ca98 SESSION: ()
> >> >> > 2024-11-20T12:26:47.910806+0300 0xc5cc3c SESSION: ()
> >> >> > 2024-11-20T12:27:35.746239+0300 0xc5ccc9 SESSION: ()
> >> >> > 2024-11-20T12:28:46.923812+0300 0xc5ce63 SESSION: ()
> >> >> > 2024-11-20T12:28:47.903066+0300 0xc5d007 SESSION: ()
> >> >> > 2024-11-20T12:29:08.063326+0300 0xc5d094 EXPORT: ()
> >> >> > di1/A2037D53
> >> >> > 2024-11-20T12:30:46.909621+0300 0xc5d96f SESSION: ()
> >> >> > 2024-11-20T12:30:47.908050+0300 0xc5db13 SESSION: ()
> >> >> > 2024-11-20T12:32:46.907649+0300 0xc5dba0 SESSION: ()
> >> >> > 2024-11-20T12:32:47.905962+0300 0xc5dd44 SESSION: ()
> >> >> > 2024-11-20T12:34:44.349348+0300 0xc5ddd1 SESSIONS: ()
> >> >> >
> >> >> > But the main problem, when I stop MDS rank 1 (without any kind of
> >> >> standby)
> >> >> > - FS hangs for all actions.
> >> >> > Is this correct? Is it possible to completely exclude rank 1 from
> >> >> > processing dir1 and not stop io when rank 1 goes down?
> >> >> > _______________________________________________
> >> >> > ceph-users mailing list -- ceph-users@xxxxxxx
> >> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >> >>
> >> >>
> >> >> _______________________________________________
> >> >> ceph-users mailing list -- ceph-users@xxxxxxx
> >> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> >> >>
> >>
> >>
> >>
> >>
>
>
>
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
