And, Eugen, try to see ceph fs status during write.
I can see next INOS, DNS and Reqs distribution:
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 active c Reqs: 127 /s 12.6k 12.5k 333 505
1 active b Reqs: 11 /s 21 24 19 1
I mean that even if I pin all top dirs (of course without repinning on
next levels) to rank 1 - I see some amount of Reqs on rank 1.
вт, 26 нояб. 2024 г. в 12:01, Александр Руденко <a.rudikk@xxxxxxxxx>:
Hm, the same test worked for me with version 16.2.13... I
mean, I only
do a few writes from a single client, so this may be an
invalid test,
but I don't see any interruption.
I tried many times and I'm sure that my test is correct.
Yes, write can be active for some time after rank 1 went down,
maybe tens seconds. And listing files (ls) can work some time for
dirs which were listed before rank down, but only fews seconds.
Before shutdown rank 1 I run write in this way:
while true; do dd if=/dev/vda of=/cephfs-mount/dir1/`uuidgen`
count=1 oflag=direct; sleep 0.000001; done
Maybe it depends on the RPS...
пт, 22 нояб. 2024 г. в 14:48, Eugen Block <eblock@xxxxxx>:
Hm, the same test worked for me with version 16.2.13... I
mean, I only
do a few writes from a single client, so this may be an
invalid test,
but I don't see any interruption.
Zitat von Eugen Block <eblock@xxxxxx>:
> I just tried to reproduce the behaviour but failed to do so.
I have
> a Reef (18.2.2) cluster with multi-active MDS. Don't mind the
> hostnames, this cluster was deployed with Nautilus.
>
> # mounted the FS
> mount -t ceph nautilus:/ /mnt -o
> name=admin,secret=****,mds_namespace=secondfs
>
> # created and pinned directories
> nautilus:~ # mkdir /mnt/dir1
> nautilus:~ # mkdir /mnt/dir2
>
> nautilus:~ # setfattr -n ceph.dir.pin -v 0 /mnt/dir1
> nautilus:~ # setfattr -n ceph.dir.pin -v 0 /mnt/dir2
>
> I stopped all standby daemons while writing into /mnt/dir1,
then I
> also stopped rank 1. But the writes were not interrupted
(until I
> stopped them). You're on Pacific, I'll see if I can
reproduce it
> there.
>
> Zitat von Александр Руденко <a.rudikk@xxxxxxxxx>:
>
>>>
>>> Can you show the entire 'ceph fs status' output? Any maybe
also 'ceph
>>> fs dump'?
>>
>>
>> Nothing special, just smoll test cluster.
>> fs1 - 10 clients
>> ===
>> RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
>> 0 active a Reqs: 0 /s 18.7k 18.4k 351 513
>> 1 active b Reqs: 0 /s 21 24 16 1
>> POOL TYPE USED AVAIL
>> fs1_meta metadata 116M 3184G
>> fs1_data data 23.8G 3184G
>> STANDBY MDS
>> c
>>
>>
>> fs dump
>>
>> e48
>> enable_multiple, ever_enabled_multiple: 1,1
>> default compat: compat={},rocompat={},incompat={1=base
v0.20,2=client
>> writeable ranges,3=default file layouts on dirs,4=dir inode
in separate
>> object,5=mds uses versioned encoding,6=dirfrag is stored in
omap,8=no
>> anchor table,9=file layout v2,10=snaprealm v2}
>> legacy client fscid: 1
>>
>> Filesystem 'fs1' (1)
>> fs_name fs1
>> epoch 47
>> flags 12
>> created 2024-10-15T18:55:10.905035+0300
>> modified 2024-11-21T10:55:12.688598+0300
>> tableserver 0
>> root 0
>> session_timeout 60
>> session_autoclose 300
>> max_file_size 1099511627776
>> required_client_features {}
>> last_failure 0
>> last_failure_osd_epoch 943
>> compat compat={},rocompat={},incompat={1=base
v0.20,2=client writeable
>> ranges,3=default file layouts on dirs,4=dir inode in
separate object,5=mds
>> uses versioned encoding,6=dirfrag is stored in omap,7=mds
uses inline
>> data,8=no anchor table,9=file layout v2,10=snaprealm v2}
>> max_mds 2
>> in 0,1
>> up {0=12200812,1=11974933}
>> failed
>> damaged
>> stopped
>> data_pools [7]
>> metadata_pool 6
>> inline_data disabled
>> balancer
>> standby_count_wanted 1
>> [mds.a{0:12200812} state up:active seq 13 addr [v2:
>> 10.7.1.115:6842/1955635987,v1:10.7.1.115:6843/1955635987
<http://10.7.1.115:6842/1955635987,v1:10.7.1.115:6843/1955635987>]
compat
>> {c=[1],r=[1],i=[7ff]}]
>> [mds.b{1:11974933} state up:active seq 5 addr [v2:
>> 10.7.1.116:6840/536741454,v1:10.7.1.116:6841/536741454
<http://10.7.1.116:6840/536741454,v1:10.7.1.116:6841/536741454>]
compat
>> {c=[1],r=[1],i=[7ff]}]
>>
>>
>> Standby daemons:
>>
>> [mds.c{-1:11704322} state up:standby seq 1 addr [v2:
>> 10.7.1.117:6848/84247504,v1:10.7.1.117:6849/84247504
<http://10.7.1.117:6848/84247504,v1:10.7.1.117:6849/84247504>]
compat
>> {c=[1],r=[1],i=[7ff]}]
>>
>> чт, 21 нояб. 2024 г. в 11:36, Eugen Block <eblock@xxxxxx>:
>>
>>> I'm not aware of any hard limit for the number of
Filesystems, but
>>> that doesn't really mean very much. IIRC, last week during
a Clyso
>>> talk at Eventbrite I heard someone say that they deployed
around 200
>>> Filesystems or so, I don't remember if it was a production
environment
>>> or just a lab environment. I assume that you would
probably be limited
>>> by the number of OSDs/PGs rather than by the number of
Filesystems,
>>> 200 Filesystems require at least 400 pools. But maybe
someone else has
>>> more experience in scaling CephFS that way. What we did
was to scale
>>> the number of active MDS daemons for one CephFS. I believe
in the end
>>> the customer had 48 MDS daemons on three MDS servers, 16
of them were
>>> active with directory pinning, at that time they had 16
standby-replay
>>> and 16 standby daemons. But it turned out that
standby-replay didn't
>>> help their use case, so we disabled standby-replay.
>>>
>>> Can you show the entire 'ceph fs status' output? Any maybe
also 'ceph
>>> fs dump'?
>>>
>>> Zitat von Александр Руденко <a.rudikk@xxxxxxxxx>:
>>>
>>>>>
>>>>> Just for testing purposes, have you tried pinning rank 1
to some other
>>>>> directory? Does it still break the CephFS if you stop it?
>>>>
>>>>
>>>> Yes, nothing changed.
>>>>
>>>> It's no problem that FS hangs when one of the ranks goes
down, we will
>>> have
>>>> standby-reply for all ranks. I don't like that rank which
is not pinned
>>> to
>>>> some dir handled some io of this dir or from clients
which work with this
>>>> dir.
>>>> I mean that I can't robustly and fully separate client IO
by ranks.
>>>>
>>>> Would it be an option to rather use multiple Filesystems
instead of
>>>>> multi-active for one CephFS?
>>>>
>>>>
>>>> Yes, it's an option. But it is much more complicated in
our case. Btw, do
>>>> you know how many different FS can be created in one
cluster? Maybe you
>>>> know some potential problems with 100-200 FSs in one cluster?
>>>>
>>>> ср, 20 нояб. 2024 г. в 17:50, Eugen Block <eblock@xxxxxx>:
>>>>
>>>>> Ah, I misunderstood, I thought you wanted an even
distribution across
>>>>> both ranks.
>>>>> Just for testing purposes, have you tried pinning rank 1
to some other
>>>>> directory? Does it still break the CephFS if you stop
it? I'm not sure
>>>>> if you can prevent rank 1 from participating, I haven't
looked into
>>>>> all the configs in quite a while. Would it be an option
to rather use
>>>>> multiple Filesystems instead of multi-active for one CephFS?
>>>>>
>>>>> Zitat von Александр Руденко <a.rudikk@xxxxxxxxx>:
>>>>>
>>>>> > No it's not a typo. It's misleading example)
>>>>> >
>>>>> > dir1 and dir2 are pinned to rank 0, but FS and
dir1,dir2 can't work
>>>>> without
>>>>> > rank 1.
>>>>> > rank 1 is used for something when I work with this dirs.
>>>>> >
>>>>> > ceph 16.2.13, metadata balancer and policy based
balancing not used.
>>>>> >
>>>>> > ср, 20 нояб. 2024 г. в 16:33, Eugen Block <eblock@xxxxxx>:
>>>>> >
>>>>> >> Hi,
>>>>> >>
>>>>> >> > After pinning:
>>>>> >> > setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir1
>>>>> >> > setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir2
>>>>> >>
>>>>> >> is this a typo? If not, you did pin both directories
to the same
>>> rank.
>>>>> >>
>>>>> >> Zitat von Александр Руденко <a.rudikk@xxxxxxxxx>:
>>>>> >>
>>>>> >> > Hi,
>>>>> >> >
>>>>> >> > I try to distribute all top level dirs in CephFS by
different MDS
>>>>> ranks.
>>>>> >> > I have two active MDS with rank *0* and *1 *and I
have 2 top dirs
>>> like
>>>>> >> > */dir1* and* /dir2*.
>>>>> >> >
>>>>> >> > After pinning:
>>>>> >> > setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir1
>>>>> >> > setfattr -n ceph.dir.pin -v 0 /fs-mountpoint/dir2
>>>>> >> >
>>>>> >> > I can see next INOS and DNS distribution:
>>>>> >> > RANK STATE MDS ACTIVITY DNS INOS DIRS
CAPS
>>>>> >> > 0 active c Reqs: 127 /s 12.6k 12.5k
333 505
>>>>> >> > 1 active b Reqs: 11 /s 21 24
19 1
>>>>> >> >
>>>>> >> > When I write to dir1 I can see a small amount on
Reqs: in rank 1.
>>>>> >> >
>>>>> >> > Events in journal of MDS with rank 1:
>>>>> >> > cephfs-journal-tool --rank=fs1:1 event get list
>>>>> >> >
>>>>> >> > 2024-11-20T12:24:42.045056+0300 0xc5c1cb UPDATE:
>>>>> (scatter_writebehind)
>>>>> >> > A2037D53
>>>>> >> > 2024-11-20T12:24:46.935934+0300 0xc5c629 SESSION: ()
>>>>> >> > 2024-11-20T12:24:47.192012+0300 0xc5c7cd UPDATE:
(lock inest
>>>>> accounted
>>>>> >> > scatter stat update)
>>>>> >> > 2024-11-20T12:24:47.904717+0300 0xc5ca0b SESSION: ()
>>>>> >> > 2024-11-20T12:26:46.912719+0300 0xc5ca98 SESSION: ()
>>>>> >> > 2024-11-20T12:26:47.910806+0300 0xc5cc3c SESSION: ()
>>>>> >> > 2024-11-20T12:27:35.746239+0300 0xc5ccc9 SESSION: ()
>>>>> >> > 2024-11-20T12:28:46.923812+0300 0xc5ce63 SESSION: ()
>>>>> >> > 2024-11-20T12:28:47.903066+0300 0xc5d007 SESSION: ()
>>>>> >> > 2024-11-20T12:29:08.063326+0300 0xc5d094 EXPORT: ()
>>>>> >> > di1/A2037D53
>>>>> >> > 2024-11-20T12:30:46.909621+0300 0xc5d96f SESSION: ()
>>>>> >> > 2024-11-20T12:30:47.908050+0300 0xc5db13 SESSION: ()
>>>>> >> > 2024-11-20T12:32:46.907649+0300 0xc5dba0 SESSION: ()
>>>>> >> > 2024-11-20T12:32:47.905962+0300 0xc5dd44 SESSION: ()
>>>>> >> > 2024-11-20T12:34:44.349348+0300 0xc5ddd1 SESSIONS: ()
>>>>> >> >
>>>>> >> > But the main problem, when I stop MDS rank 1
(without any kind of
>>>>> >> standby)
>>>>> >> > - FS hangs for all actions.
>>>>> >> > Is this correct? Is it possible to completely
exclude rank 1 from
>>>>> >> > processing dir1 and not stop io when rank 1 goes down?
>>>>> >> > _______________________________________________
>>>>> >> > ceph-users mailing list -- ceph-users@xxxxxxx
>>>>> >> > To unsubscribe send an email to
ceph-users-leave@xxxxxxx
>>>>> >>
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> ceph-users mailing list -- ceph-users@xxxxxxx
>>>>> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>>>> >>
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>>>
