Related: Where can I find MDS numeric state references for ceph mds
set_state GID <int(state)>?
Like a dummy I accidentally upgraded to the ceph dev branch (quincy?),
and have been having nothing but trouble since. This wasn't actually
intentionally, I was trying to implement a PR which was expected to
bring my cluster back online.
--> It did bring my cluster back online, so that's good, but I failed
to recognize that by building from the master branch that I also
wouldn't be able to revert later. Whoops.
Lastly, while my MDS are online (MONs, OSDs and MGRs too.) MDSs are
never marked up:active, so my cephfs data is inaccessible.
Hoping someone can help me determine the best way to bring an MDS up:active.
mds log immediately after restarting mds:
> 2021-08-08T09:51:06.673-0600 7f7475f0d700 -1 received signal: Terminated from /sbin/init (PID: 1) UID: 0
> 2021-08-08T09:51:06.673-0600 7f7475f0d700 -1 mds.server *** got signal Terminated ***
> 2021-08-08T09:51:06.673-0600 7f7475f0d700 1 mds.server suicide! Wanted state up:standby
> 2021-08-08T09:51:06.673-0600 7f7475f0d700 5 mds.beacon.server set_want_state: up:standby -> down:dne
> 2021-08-08T09:51:06.673-0600 7f7475f0d700 5 mds.beacon.server Sending beacon down:dne seq 1368
> 2021-08-08T09:51:06.673-0600 7f7475f0d700 20 mds.beacon.server send_and_wait: awaiting 1368 for up to 1s
> 2021-08-08T09:51:06.697-0600 7f7477f11700 5 mds.beacon.server received beacon reply down:dne seq 1368 rtt 0.0240006
> 2021-08-08T09:51:09.001-0600 7f747470a700 0 ms_deliver_dispatch: unhandled message 0x55bf0599c340 mdsmap(e 1095533) v2 from mon.1 v2:192.168.2.2:3300/0
> 2021-08-08T09:51:09.413-0600 7fb01a664600 0 set uid:gid to 64045:64045 (ceph:ceph)
> 2021-08-08T09:51:09.413-0600 7fb01a664600 0 ceph version 17.0.0-6762-g0ff2e281889 (0ff2e28188997853804c97d42dd070c38127c94d) quincy (dev), process ceph-mds, pid 1688503
> 2021-08-08T09:51:09.413-0600 7fb01a664600 1 main not setting numa affinity
> 2021-08-08T09:51:09.413-0600 7fb01a664600 0 pidfile_write: ignore empty --pid-file
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server Dumping misc struct sizes:
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 96 MDSCacheObject
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 1328 CInode
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 16 elist<>::item
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 560 inode
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 632 old_inode
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 48 nest_info_t
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 40 frag_info_t
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 40 SimpleLock
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 48 ScatterLock
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 600 CDentry
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 16 elist<>::item
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 40 SimpleLock
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 1088 CDir
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 16 elist<>::item
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 232 fnode_t
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 48 nest_info_t
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 40 frag_info_t
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 320 Capability
> 2021-08-08T09:51:09.417-0600 7fb01a664600 10 mds.server 32 xlist<>::item
> 2021-08-08T09:51:09.417-0600 7fb015e05700 1 mds.server Updating MDS map to version 1095533 from mon.1
> 2021-08-08T09:51:09.417-0600 7fb013e01700 5 mds.beacon.server Sending beacon up:boot seq 1
> 2021-08-08T09:51:09.417-0600 7fb015e05700 10 mds.server my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
> 2021-08-08T09:51:09.417-0600 7fb015e05700 10 mds.server mdsmap compat compat={},rocompat={},incompat={}
> 2021-08-08T09:51:09.417-0600 7fb015e05700 10 mds.server my gid is 138479365
> 2021-08-08T09:51:09.417-0600 7fb015e05700 10 mds.server map says I am mds.-1.-1 state null
> 2021-08-08T09:51:09.417-0600 7fb015e05700 10 mds.server msgr says I am [v2:192.168.2.2:6808/1262779536,v1:192.168.2.2:6809/1262779536]
> 2021-08-08T09:51:09.417-0600 7fb015e05700 10 mds.server handle_mds_map: handling map in rankless mode
> 2021-08-08T09:51:09.441-0600 7fb013e01700 20 mds.beacon.server sender thread waiting interval 4s
> 2021-08-08T09:51:09.441-0600 7fb015e05700 10 mds.server not in map yet
> 2021-08-08T09:51:09.765-0600 7fb015e05700 1 mds.server Updating MDS map to version 1095534 from mon.1
> 2021-08-08T09:51:09.765-0600 7fb015e05700 10 mds.server my compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
> 2021-08-08T09:51:09.765-0600 7fb015e05700 10 mds.server mdsmap compat compat={},rocompat={},incompat={}
> 2021-08-08T09:51:09.765-0600 7fb015e05700 10 mds.server my gid is 138479365
> 2021-08-08T09:51:09.765-0600 7fb015e05700 10 mds.server map says I am mds.-1.0 state up:standby
> 2021-08-08T09:51:09.765-0600 7fb015e05700 10 mds.server msgr says I am [v2:192.168.2.2:6808/1262779536,v1:192.168.2.2:6809/1262779536]
> 2021-08-08T09:51:09.765-0600 7fb015e05700 10 mds.server handle_mds_map: handling map in rankless mode
> 2021-08-08T09:51:09.765-0600 7fb015e05700 1 mds.server Monitors have assigned me to become a standby.
> 2021-08-08T09:51:09.765-0600 7fb015e05700 5 mds.beacon.server set_want_state: up:boot -> up:standby
> 2021-08-08T09:51:09.777-0600 7fb018e0b700 5 mds.beacon.server received beacon reply up:boot seq 1 rtt 0.360009
> 2021-08-08T09:51:13.442-0600 7fb013e01700 5 mds.beacon.server Sending beacon up:standby seq 2
> 2021-08-08T09:51:13.442-0600 7fb013e01700 20 mds.beacon.server sender thread waiting interval 4s
> 2021-08-08T09:51:13.442-0600 7fb018e0b700 5 mds.beacon.server received beacon reply up:standby seq 2 rtt 0
> 2021-08-08T09:51:17.442-0600 7fb013e01700 5 mds.beacon.server Sending beacon up:standby seq 3
> 2021-08-08T09:51:17.442-0600 7fb013e01700 20 mds.beacon.server sender thread waiting interval 4s
> 2021-08-08T09:51:17.442-0600 7fb018e0b700 5 mds.beacon.server received beacon reply up:standby seq 3 rtt 0
> 2021-08-08T09:51:21.442-0600 7fb013e01700 5 mds.beacon.server Sending beacon up:standby seq 4
> 2021-08-08T09:51:21.442-0600 7fb013e01700 20 mds.beacon.server sender thread waiting interval 4s
> 2021-08-08T09:51:21.442-0600 7fb018e0b700 5 mds.beacon.server received beacon reply up:standby seq 4 rtt 0
> 2021-08-08T09:51:25.442-0600 7fb013e01700 5 mds.beacon.server Sending beacon up:standby seq 5
> 2021-08-08T09:51:25.442-0600 7fb013e01700 20 mds.beacon.server sender thread waiting interval 4s
> 2021-08-08T09:51:25.442-0600 7fb018e0b700 5 mds.beacon.server received beacon reply up:standby seq 5 rtt 0
> 2021-08-08T09:51:29.442-0600 7fb013e01700 5 mds.beacon.server Sending beacon up:standby seq 6
> 2021-08-08T09:51:29.442-0600 7fb013e01700 20 mds.beacon.server sender thread waiting interval 4s
> 2021-08-08T09:51:29.442-0600 7fb018e0b700 5 mds.beacon.server received beacon reply up:standby seq 6 rtt 0
cycles on this forever, never marked up.
And this odd stuff out of the mon log:
>
> 2021-08-08T10:04:57.136-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/566717310' entity='client.admin' cmd=[{"format":"json","prefix":"df"}]: dispatch
> 2021-08-08T10:04:58.220-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1
> 2021-08-08T10:04:58.220-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.20:0/2526429427' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch
> 2021-08-08T10:04:58.912-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"status"} v 0) v1
> 2021-08-08T10:04:58.912-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/3325895601' entity='client.admin' cmd=[{"format":"json","prefix":"status"}]: dispatch
> 2021-08-08T10:04:58.920-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"detail":"detail","format":"json","prefix":"health"} v 0) v1
> 2021-08-08T10:04:58.920-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/3325895601' entity='client.admin' cmd=[{"detail":"detail","format":"json","prefix":"health"}]: dispatch
> 2021-08-08T10:04:58.928-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"mon dump"} v 0) v1
> 2021-08-08T10:04:58.928-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/3325895601' entity='client.admin' cmd=[{"format":"json","prefix":"mon dump"}]: dispatch
> 2021-08-08T10:04:58.932-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"mgr dump"} v 0) v1
> 2021-08-08T10:04:58.932-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/3325895601' entity='client.admin' cmd=[{"format":"json","prefix":"mgr dump"}]: dispatch
> 2021-08-08T10:04:59.372-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1
> 2021-08-08T10:04:59.372-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.4:0/403800046' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch
> 2021-08-08T10:04:59.652-0600 7fd991d78700 1 mon.rog@3(peon).osd e1101076 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 71303168 full_alloc: 71303168 kv_alloc: 872415232
> 2021-08-08T10:04:59.928-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1
> 2021-08-08T10:04:59.928-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.4:0/1216981870' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch
> 2021-08-08T10:05:04.652-0600 7fd991d78700 1 mon.rog@3(peon).osd e1101076 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 71303168 full_alloc: 71303168 kv_alloc: 872415232
> 2021-08-08T10:05:04.780-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1
> 2021-08-08T10:05:04.780-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.6:0/4082867240' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch
> 2021-08-08T10:05:05.356-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1
> 2021-08-08T10:05:05.356-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.6:0/388331640' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch
> 2021-08-08T10:05:05.372-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"df"} v 0) v1
> 2021-08-08T10:05:05.372-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.6:0/3652273871' entity='client.admin' cmd=[{"format":"json","prefix":"df"}]: dispatch
> 2021-08-08T10:05:05.924-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"status","format":"json"} v 0) v1
> 2021-08-08T10:05:05.924-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/2771394330' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch
> 2021-08-08T10:05:05.928-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"health","detail":"detail","format":"json"} v 0) v1
> 2021-08-08T10:05:05.928-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/2771394330' entity='client.admin' cmd=[{"prefix":"health","detail":"detail","format":"json"}]: dispatch
> 2021-08-08T10:05:05.936-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"mon dump","format":"json"} v 0) v1
> 2021-08-08T10:05:05.936-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/2771394330' entity='client.admin' cmd=[{"prefix":"mon dump","format":"json"}]: dispatch
> 2021-08-08T10:05:05.940-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"mgr dump"} v 0) v1
> 2021-08-08T10:05:05.940-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/2771394330' entity='client.admin' cmd=[{"format":"json","prefix":"mgr dump"}]: dispatch
> 2021-08-08T10:05:06.908-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"mon metadata"} v 0) v1
> 2021-08-08T10:05:06.908-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.4:0/506622373' entity='client.admin' cmd=[{"format":"json","prefix":"mon metadata"}]: dispatch
> 2021-08-08T10:05:06.908-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"mgr metadata","format":"json"} v 0) v1
> 2021-08-08T10:05:06.908-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.4:0/506622373' entity='client.admin' cmd=[{"prefix":"mgr metadata","format":"json"}]: dispatch
> 2021-08-08T10:05:06.912-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"mds metadata","format":"json"} v 0) v1
> 2021-08-08T10:05:06.912-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.4:0/506622373' entity='client.admin' cmd=[{"prefix":"mds metadata","format":"json"}]: dispatch
> 2021-08-08T10:05:06.912-0600 7fd98f573700 1 mon.rog@3(peon).mds e1095535 all = 1
> 2021-08-08T10:05:06.912-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"osd metadata"} v 0) v1
> 2021-08-08T10:05:06.912-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.4:0/506622373' entity='client.admin' cmd=[{"format":"json","prefix":"osd metadata"}]: dispatch
> 2021-08-08T10:05:07.612-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1
> 2021-08-08T10:05:07.612-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/1568006528' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch
> 2021-08-08T10:05:07.688-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1
> 2021-08-08T10:05:07.688-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/3150627131' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch
> 2021-08-08T10:05:08.596-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"df"} v 0) v1
> 2021-08-08T10:05:08.596-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.20:0/3181651427' entity='client.admin' cmd=[{"format":"json","prefix":"df"}]: dispatch
> 2021-08-08T10:05:08.924-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1
> 2021-08-08T10:05:08.924-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.4:0/4143418844' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch
> 2021-08-08T10:05:09.120-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"df"} v 0) v1
> 2021-08-08T10:05:09.120-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.4:0/733056629' entity='client.admin' cmd=[{"format":"json","prefix":"df"}]: dispatch
> 2021-08-08T10:05:09.656-0600 7fd991d78700 1 mon.rog@3(peon).osd e1101076 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 71303168 full_alloc: 71303168 kv_alloc: 872415232
> 2021-08-08T10:05:12.916-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"status","format":"json"} v 0) v1
> 2021-08-08T10:05:12.916-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/3083014270' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch
> 2021-08-08T10:05:12.920-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","detail":"detail","prefix":"health"} v 0) v1
> 2021-08-08T10:05:12.920-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/3083014270' entity='client.admin' cmd=[{"format":"json","detail":"detail","prefix":"health"}]: dispatch
> 2021-08-08T10:05:12.932-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"mon dump"} v 0) v1
> 2021-08-08T10:05:12.932-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/3083014270' entity='client.admin' cmd=[{"format":"json","prefix":"mon dump"}]: dispatch
> 2021-08-08T10:05:12.932-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"mgr dump","format":"json"} v 0) v1
> 2021-08-08T10:05:12.932-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/3083014270' entity='client.admin' cmd=[{"prefix":"mgr dump","format":"json"}]: dispatch
> 2021-08-08T10:05:14.656-0600 7fd991d78700 1 mon.rog@3(peon).osd e1101076 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 71303168 full_alloc: 71303168 kv_alloc: 872415232
> 2021-08-08T10:05:14.664-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"df"} v 0) v1
> 2021-08-08T10:05:14.664-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.6:0/3518250893' entity='client.admin' cmd=[{"format":"json","prefix":"df"}]: dispatch
> 2021-08-08T10:05:14.864-0600 7fd98dd70700 0 --1- [v2:192.168.2.6:3300/0,v1:192.168.2.6:6789/0] >> conn(0x561e07c38400 0x561e14765000 :6789 s=ACCEPTING pgs=0 cs=0 l=0).handle_client_banner accept peer addr is really - (socket is v1:192.168.2.6:55784/0)
> 2021-08-08T10:05:15.488-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"df"} v 0) v1
> 2021-08-08T10:05:15.488-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.6:0/4145195535' entity='client.admin' cmd=[{"format":"json","prefix":"df"}]: dispatch
> 2021-08-08T10:05:16.452-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1
> 2021-08-08T10:05:16.452-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/3308082453' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch
> 2021-08-08T10:05:17.372-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"df"} v 0) v1
> 2021-08-08T10:05:17.372-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.4:0/945662900' entity='client.admin' cmd=[{"format":"json","prefix":"df"}]: dispatch
> 2021-08-08T10:05:17.776-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1
> 2021-08-08T10:05:17.776-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.20:0/3481376298' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch
> 2021-08-08T10:05:17.852-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1
> 2021-08-08T10:05:17.852-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.20:0/2868178156' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch
> 2021-08-08T10:05:17.944-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"df","format":"json"} v 0) v1
> 2021-08-08T10:05:17.944-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.4:0/2212650306' entity='client.admin' cmd=[{"prefix":"df","format":"json"}]: dispatch
> 2021-08-08T10:05:18.944-0600 7fd98dd70700 0 --1- [v2:192.168.2.6:3300/0,v1:192.168.2.6:6789/0] >> conn(0x561e0593f800 0x561e07c57000 :6789 s=ACCEPTING pgs=0 cs=0 l=0).handle_client_banner accept peer addr is really - (socket is v1:192.168.2.20:60454/0)
> 2021-08-08T10:05:19.656-0600 7fd991d78700 1 mon.rog@3(peon).osd e1101076 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 71303168 full_alloc: 71303168 kv_alloc: 872415232
> 2021-08-08T10:05:19.908-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"status","format":"json"} v 0) v1
> 2021-08-08T10:05:19.908-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/3424553700' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch
> 2021-08-08T10:05:19.912-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"prefix":"health","detail":"detail","format":"json"} v 0) v1
> 2021-08-08T10:05:19.912-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/3424553700' entity='client.admin' cmd=[{"prefix":"health","detail":"detail","format":"json"}]: dispatch
> 2021-08-08T10:05:19.920-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"mon dump"} v 0) v1
> 2021-08-08T10:05:19.920-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/3424553700' entity='client.admin' cmd=[{"format":"json","prefix":"mon dump"}]: dispatch
> 2021-08-08T10:05:19.932-0600 7fd98f573700 0 mon.rog@3(peon) e137 handle_command mon_command({"format":"json","prefix":"mgr dump"} v 0) v1
> 2021-08-08T10:05:19.932-0600 7fd98f573700 0 log_channel(audit) log [DBG] : from='client.? 192.168.2.2:0/3424553700' entity='client.admin' cmd=[{"format":"json","prefix":"mgr dump"}]: dispatch
This in particular looks weird to me, as 192.168.2.20 is a different node:
> [v2:192.168.2.6:3300/0,v1:192.168.2.6:6789/0] >> conn(0x561e0593f800 0x561e07c57000 :6789 s=ACCEPTING pgs=0 cs=0 l=0).handle_client_banner accept peer addr is really - (socket is v1:192.168.2.20:60454/0)
As far as I can tell, no cephfs or mds settings seem to help. Number
of ranks, standy-active or not, cephx or not, different networks,
recreating mgrs mons or mds, etc.
I did however notice this and am hoping someone can confirm if normal,
or if I am about to go on a goose chase. Out of this block of the MDS
log:
2021-08-10T09:31:09.484-0600 7ffa894fc700 1 mds.rog Updating MDS map
to version 1095550 from mon.2
2021-08-10T09:31:09.484-0600 7ffa894fc700 10 mds.rog my compat
compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds
uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
2021-08-10T09:31:09.484-0600 7ffa894fc700 10 mds.rog mdsmap compat
compat={},rocompat={},incompat={}
2021-08-10T09:31:09.484-0600 7ffa894fc700 10 mds.rog my gid is 139597028
2021-08-10T09:31:09.484-0600 7ffa894fc700 10 mds.rog map says I am
mds.-1.-1 state null
2021-08-10T09:31:09.484-0600 7ffa894fc700 10 mds.rog msgr says I am
[v2:192.168.10.50:6800/1353942242,v1:192.168.10.50:6801/1353942242]
2021-08-10T09:31:09.484-0600 7ffa894fc700 10 mds.rog handle_mds_map:
handling map in rankless mode
2021-08-10T09:31:09.484-0600 7ffa894fc700 10 mds.rog not in map yet
2021-08-10T09:31:10.000-0600 7ffa894fc700 1 mds.rog Updating MDS map
to version 1095551 from mon.2
2021-08-10T09:31:10.000-0600 7ffa894fc700 10 mds.rog my compat
compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds
uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
2021-08-10T09:31:10.000-0600 7ffa894fc700 10 mds.rog mdsmap compat
compat={},rocompat={},incompat={}
2021-08-10T09:31:10.000-0600 7ffa894fc700 10 mds.rog my gid is 139597028
2021-08-10T09:31:10.000-0600 7ffa894fc700 10 mds.rog map says I am
mds.-1.0 state up:standby
2021-08-10T09:31:10.000-0600 7ffa894fc700 10 mds.rog msgr says I am
[v2:192.168.10.50:6800/1353942242,v1:192.168.10.50:6801/1353942242]
2021-08-10T09:31:10.000-0600 7ffa894fc700 10 mds.rog handle_mds_map:
handling map in rankless mode
2021-08-10T09:31:10.000-0600 7ffa894fc700 1 mds.rog Monitors have
assigned me to become a standby.
2021-08-10T09:31:10.000-0600 7ffa894fc700 5 mds.beacon.rog
set_want_state: up:boot -> up:standby
These Lines:
2021-08-10T09:31:09.484-0600 7ffa894fc700 10 mds.rog my compat
compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds
uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
2021-08-10T09:31:09.484-0600 7ffa894fc700 10 mds.rog mdsmap compat
compat={},rocompat={},incompat={}
---> Looks like a difference between the mdsmap and MDS's "incompat={"
setting. Is one meant to be `incompat={}` ?
This Line:
2021-08-10T09:31:09.484-0600 7ffa894fc700 10 mds.rog map says I am
mds.-1.-1 state null
---> Is mds.-1.-1 normal?
This Line:
2021-08-10T09:31:10.000-0600 7ffa894fc700 10 mds.rog handle_mds_map:
handling map in rankless mode
---> Is "handling map in rankless mode" normal?
I am hopeful to recover my cephfs data by whatever means makes the most sense.
Eventually I intend to create a seperate temporary ceph cluster, and
migrate my data back to Pacific but really don't want to abandon this
data if I can avoid it.
Help!
~Josh
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
