We are experiencing the same when upgrading to 16.2.6 with cephadm.
I tried
ceph fs set cephfs max_mds 1
ceph fs set cephfs allow_standby_replay false
, but still all MDS goes to standby. It seems all ranks are marked failed. Do we have a way to clear this flag?
Please help. Our cluster is down. Thanks.
# ceph fs status
cephfs - 0 clients
======
RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS
0 failed
1 failed
POOL TYPE USED AVAIL
cephfs.cephfs.meta metadata 114G 404G
cephfs.cephfs.data data 84.9T 17.6T
cephfs.cephfs.data_ssd data 0 606G
cephfs.cephfs.data_mixed data 9879G 404G
STANDBY MDS
cephfs.gpu023.aetiph
cephfs.gpu018.ovxvoz
cephfs.gpu006.ddpekw
cephfs.gpu024.rpfbnh
MDS version: ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
# ceph fs dump
e41422
enable_multiple, ever_enabled_multiple: 0,1
default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 2
Filesystem 'cephfs' (2)
fs_name cephfs
epoch 41422
flags 12
created 2020-09-15T04:10:53.585782+0000
modified 2021-09-17T15:05:26.239956+0000
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
required_client_features {}
last_failure 0
last_failure_osd_epoch 43315
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
in 0,1
up {}
failed 0,1
damaged
stopped
data_pools [5,13,16]
metadata_pool 4
inline_data disabled
balancer
standby_count_wanted 1
Standby daemons:
[mds.cephfs.gpu023.aetiph{-1:7908668} state up:standby seq 1 join_fscid=2 addr [v2:202.38.247.186:6800/3495351337,v1:202.38.247.186:6801/3495351337] compat {c=[1],r=[1],i=[7ff]}]
[mds.cephfs.gpu018.ovxvoz{ffffffff:78cdb8} state up:standby seq 1 join_fscid=2 addr [v2:202.38.247.181:1a90/94680caa,v1:202.38.247.181:1a91/94680caa] compat {c=[1],r=[1],i=[7ff]}]
[mds.cephfs.gpu006.ddpekw{ffffffff:78f84f} state up:standby seq 1 join_fscid=2 addr [v2:202.38.247.175:1a90/fdd0fd1a,v1:202.38.247.175:1a91/fdd0fd1a] compat {c=[1],r=[1],i=[7ff]}]
[mds.cephfs.gpu024.rpfbnh{ffffffff:78fc4e} state up:standby seq 1 join_fscid=2 addr [v2:202.38.247.187:1a90/4e2e69dc,v1:202.38.247.187:1a91/4e2e69dc] compat {c=[1],r=[1],i=[7ff]}]
dumped fsmap epoch 41422
发件人: Patrick Donnelly<mailto:pdonnell@xxxxxxxxxx>
发送时间: 2021年9月17日 22:42
收件人: Eric Dold<mailto:dold.eric@xxxxxxxxx>
抄送: ceph-users<mailto:ceph-users@xxxxxxx>
主题: Re: Cephfs - MDS all up:standby, not becoming up:active
On Fri, Sep 17, 2021 at 8:54 AM Eric Dold <dold.eric@xxxxxxxxx> wrote:
>
> Hi,
>
> I get the same after upgrading to 16.2.6. All mds daemons are standby.
>
> After setting
> ceph fs set cephfs max_mds 1
> ceph fs set cephfs allow_standby_replay false
> the mds still wants to be standby.
>
> 2021-09-17T14:40:59.371+0200 7f810a58f600 0 ceph version 16.2.6
> (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable), process
> ceph-mds, pid 7113
> 2021-09-17T14:40:59.371+0200 7f810a58f600 1 main not setting numa affinity
> 2021-09-17T14:40:59.371+0200 7f810a58f600 0 pidfile_write: ignore empty
> --pid-file
> 2021-09-17T14:40:59.375+0200 7f8105cf1700 1 mds.ceph3 Updating MDS map to
> version 226251 from mon.0
> 2021-09-17T14:41:00.455+0200 7f8105cf1700 1 mds.ceph3 Updating MDS map to
> version 226252 from mon.0
> 2021-09-17T14:41:00.455+0200 7f8105cf1700 1 mds.ceph3 Monitors have
> assigned me to become a standby.
>
> setting add_incompat 1 does also not work:
> # ceph fs compat cephfs add_incompat 1
> Error EINVAL: adding a feature requires a feature string
>
> Any ideas?
Please share `ceph fs dump`.
--
Patrick Donnelly, Ph.D.
He / Him / His
Principal Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
