hi there,
we are struggling with the creation of cephfs-snapshots: doing so
reproducible causes a failover of our metadata servers. afterwards, the
demoted mds servers won't be available as standby servers and the mds
daemons on these machines have to be manually restarted. more often than
we wish, the failover fails altogether, resulting in an unresponsive cephfs.
this is with mimic 13.2.6 and a single cephfs. we are running 4 mds
servers with plenty of cpu and ram ressources in a multi-active setup
with 2 active and 2 standby mds's:
mds: ceph-fs-2/2/2 up
{0=juju-d0f708-3-lxd-1=up:active,1=juju-d0f708-5-lxd-1=up:active}, 2
up:standby-replay
is the transition from active to the standby mds servers intended? and
if not: how can we prevent those?
we could live with the failover if the ex-active mds's would still be a
part of the cluster afterwards, but this is not the case. on top of
that, the failover is not 100% reliable - if it fails, the newly active
mds's exhibit the same symptoms as the failed ones: they just sit around
complaining about "MDS internal heartbeat is not healthy!".
strace'ing the mds processes on the ex-active mds shows that they are
mostly waiting for some futex'es to become available. we also found that
the issue gets alleviated a bit by rising mds_cache_memory_limit from
its default of 1gb to 32gb - in this case, the failover has a higher
chance to succeed.
below are some logs from a successful failover - juju-d0f708-9-lxd-1 and
juju-d0f708-10-lxd-1 were the active mds's and juju-d0f708-3-lxd-1 and
juju-d0f708-5-lxd-1 the standbys. sorry for being very verbose, i don't
want to withhold any information that might be necessary to debug this
issue … if it helps, i can also provide the logs for
juju-d0f708-10-lxd-1 and juju-d0f708-5-lxd-1 and the output of ceph
daemon perf dump on all mds's before and after the issue occurs.
thank you very much & with kind regards,
t.
--- logs ---
ceph-mgr.log on a mon:
2019-08-20 09:18:23.642 7f5679639700 0 ms_deliver_dispatch: unhandled
message 0x5614f2184000 mgrreport(mds.juju-d0f708-9-lxd-1 +0-0 packed
1374) v6 from mds.0 172.28.9.20:6800/2237168008
2019-08-20 09:18:23.646 7f5684835700 1 mgr finish mon failed to return
metadata for mds.juju-d0f708-9-lxd-1: (22) Invalid argument
2019-08-20 09:18:55.781 7f5679639700 0 ms_deliver_dispatch: unhandled
message 0x5614f256e700 mgrreport(mds.juju-d0f708-10-lxd-1 +0-0 packed
1374) v6 from mds.1 172.28.9.21:6800/2008779502
2019-08-20 09:18:55.781 7f5684835700 1 mgr finish mon failed to return
metadata for mds.juju-d0f708-10-lxd-1: (22) Invalid argument
2019-08-20 09:21:26.562 7f5679639700 0 ms_deliver_dispatch: unhandled
message 0x5614f8780a00 mgrreport(mds.juju-d0f708-10-lxd-1 +0-0 packed 6)
v6 from mds.? 172.28.9.21:6800/1319885328
2019-08-20 09:21:26.562 7f5684835700 1 mgr finish mon failed to return
metadata for mds.juju-d0f708-10-lxd-1: (22) Invalid argument
2019-08-20 09:21:27.558 7f5679639700 0 ms_deliver_dispatch: unhandled
message 0x5614f5666a00 mgrreport(mds.juju-d0f708-10-lxd-1 +0-0 packed 6)
v6 from mds.? 172.28.9.21:6800/1319885328
2019-08-20 09:21:27.562 7f5684835700 1 mgr finish mon failed to return
metadata for mds.juju-d0f708-10-lxd-1: (22) Invalid argument
2019-08-20 09:21:28.558 7f5679639700 0 ms_deliver_dispatch: unhandled
message 0x5614f2e3ee00 mgrreport(mds.juju-d0f708-10-lxd-1 +0-0 packed 6)
v6 from mds.? 172.28.9.21:6800/1319885328
2019-08-20 09:21:28.562 7f5684835700 1 mgr finish mon failed to return
metadata for mds.juju-d0f708-10-lxd-1: (22) Invalid argument
2019-08-20 09:21:29.558 7f5679639700 0 ms_deliver_dispatch: unhandled
message 0x5614efa71880 mgrreport(mds.juju-d0f708-10-lxd-1 +0-0 packed 6)
v6 from mds.? 172.28.9.21:6800/1319885328
[… more of these]
ceph-mds.log on juju-d0f708-9-lxd-1 (ex active rank 0):
2019-08-20 09:17:44.824 7f7d3a138700 5 mds.beacon.juju-d0f708-9-lxd-1
Sending beacon up:active seq 59
2019-08-20 09:17:44.824 7f7d3fb2b700 5 mds.beacon.juju-d0f708-9-lxd-1
received beacon reply up:active seq 59 rtt 0
2019-08-20 09:17:45.020 7f7d3db27700 4 mds.0.server
handle_client_request client_request(client.89193:898963 getattr
pAsLsXsFs #0x1000000221e 2019-08-20 09:17:45
.021386 caller_uid=0, caller_gid=0{}) v2
[… lots of these …]
2019-08-20 09:17:45.092 7f7d36130700 5 mds.0.log _submit_thread
109124009784~1190 : EUpdate cap update [metablob 0x1000000004c, 1 dirs]
2019-08-20 09:17:45.092 7f7d3db27700 4 mds.0.server
handle_client_request client_request(client.49306:901053 getattr
pAsLsXsFs #0x1000000221f 2019-08-20 09:17:45
.093085 caller_uid=0, caller_gid=0{}) v2
[… lots of these …]
2019-08-20 09:17:45.260 7f7d36130700 5 mds.0.log _submit_thread
109124010994~107 : ETableServer snaptable prepare reqid 2 mds.0 tid 98
version 98 mutation=43 bytes
2019-08-20 09:17:45.264 7f7d36130700 5 mds.0.log _submit_thread
109124011121~11940 : EUpdate mksnap [metablob 0x1, 1 dirs table_tids=^A,98]
2019-08-20 09:17:45.272 7f7d36130700 5 mds.0.log _submit_thread
109124023081~64 : ETableServer snaptable commit tid 98 version 99
2019-08-20 09:17:45.272 7f7d3db27700 3 mds.0.server
handle_client_session client_session(request_renewcaps seq 4723) from
client.89196
[… lots of these]
2019-08-20 09:17:47.556 7f7d3db27700 4 mds.0.server
handle_client_request client_request(client.49300:968447 getattr
pAsLsXsFs #0x10000002153 2019-08-20 09:17:47.557632 caller_uid=0,
caller_gid=0{}) v2
2019-08-20 09:17:47.564 7f7d3db27700 4 mds.0.server
handle_client_request client_request(client.12139:3732538 getattr
pAsLsXsFs #0x10000002b59 2019-08-20 09:17:47.563937 caller_uid=0,
caller_gid=0{}) v2
2019-08-20 09:17:47.564 7f7d36130700 5 mds.0.log _submit_thread
109124038898~1228 : EUpdate cap update [metablob 0x1000000000a, 1 dirs]
2019-08-20 09:17:48.824 7f7d3a138700 5 mds.beacon.juju-d0f708-9-lxd-1
Sending beacon up:active seq 60
2019-08-20 09:17:48.824 7f7d3fb2b700 5 mds.beacon.juju-d0f708-9-lxd-1
received beacon reply up:active seq 60 rtt 0
2019-08-20 09:17:52.824 7f7d3a138700 5 mds.beacon.juju-d0f708-9-lxd-1
Sending beacon up:active seq 61
2019-08-20 09:17:52.824 7f7d3fb2b700 5 mds.beacon.juju-d0f708-9-lxd-1
received beacon reply up:active seq 61 rtt 0
2019-08-20 09:17:56.824 7f7d3a138700 5 mds.beacon.juju-d0f708-9-lxd-1
Sending beacon up:active seq 62
2019-08-20 09:17:56.824 7f7d3fb2b700 5 mds.beacon.juju-d0f708-9-lxd-1
received beacon reply up:active seq 62 rtt 0
2019-08-20 09:18:00.824 7f7d3a138700 5 mds.beacon.juju-d0f708-9-lxd-1
Sending beacon up:active seq 63
2019-08-20 09:18:00.824 7f7d3fb2b700 5 mds.beacon.juju-d0f708-9-lxd-1
received beacon reply up:active seq 63 rtt 0
2019-08-20 09:18:04.824 7f7d3a138700 1 heartbeat_map is_healthy
'MDSRank' had timed out after 15
2019-08-20 09:18:04.824 7f7d3a138700 0 mds.beacon.juju-d0f708-9-lxd-1
Skipping beacon heartbeat to monitors (last acked 4s ago); MDS internal
heartbeat is not healthy!
[… more of these]
ceph-mds.log on juju-d0f708-3-lxd-1 (ex standby rank 0):
2019-08-20 09:17:57.416 7f6c70964700 5 mds.beacon.juju-d0f708-3-lxd-1
Sending beacon up:standby-replay seq 34
2019-08-20 09:17:57.420 7f6c76357700 5 mds.beacon.juju-d0f708-3-lxd-1
received beacon reply up:standby-replay seq 34 rtt 0.00399997
2019-08-20 09:17:58.168 7f6c71165700 5 mds.0.0 Restarting replay as
standby-replay
2019-08-20 09:17:58.172 7f6c6d95e700 2 mds.0.0 boot_start 2: replaying
mds log
2019-08-20 09:17:58.172 7f6c6d95e700 5 mds.0.0 Finished replaying
journal as standby-replay
2019-08-20 09:17:59.172 7f6c71165700 5 mds.0.0 Restarting replay as
standby-replay
[… more of these …]
2019-08-20 09:18:17.420 7f6c70964700 5 mds.beacon.juju-d0f708-3-lxd-1
Sending beacon up:standby-replay seq 39
2019-08-20 09:18:17.420 7f6c76357700 5 mds.beacon.juju-d0f708-3-lxd-1
received beacon reply up:standby-replay seq 39 rtt 0
2019-08-20 09:18:18.216 7f6c71165700 5 mds.0.0 Restarting replay as
standby-replay
2019-08-20 09:18:18.220 7f6c6d95e700 2 mds.0.0 boot_start 2: replaying
mds log
2019-08-20 09:18:18.220 7f6c6d95e700 5 mds.0.0 Finished replaying
journal as standby-replay
2019-08-20 09:18:18.756 7f6c74353700 4 mds.0.0 handle_osd_map epoch
7445, 0 new blacklist entries
2019-08-20 09:18:18.776 7f6c74353700 1 mds.juju-d0f708-3-lxd-1 Updating
MDS map to version 6084 from mon.0
2019-08-20 09:18:18.776 7f6c74353700 1 mds.0.6084 handle_mds_map i am
now mds.0.6084
2019-08-20 09:18:18.776 7f6c74353700 1 mds.0.6084 handle_mds_map state
change up:standby-replay --> up:replay
2019-08-20 09:18:18.776 7f6c74353700 5 mds.beacon.juju-d0f708-3-lxd-1
set_want_state: up:standby-replay -> up:replay
2019-08-20 09:18:19.220 7f6c71165700 5 mds.0.6084 Restarting replay as
standby-replay
2019-08-20 09:18:19.240 7f6c6d95e700 2 mds.0.6084 boot_start 2:
replaying mds log
2019-08-20 09:18:19.240 7f6c6d95e700 5 mds.0.6084 Finished replaying
journal as standby-replay
2019-08-20 09:18:19.240 7f6c6d95e700 1 mds.0.6084
standby_replay_restart (final takeover pass)
2019-08-20 09:18:19.240 7f6c6d95e700 1 mds.0.6084 opening purge_queue
(async)
2019-08-20 09:18:19.240 7f6c6d95e700 4 mds.0.purge_queue open: opening
2019-08-20 09:18:19.240 7f6c6d95e700 1 mds.0.6084 opening
open_file_table (async)
2019-08-20 09:18:19.240 7f6c6d95e700 2 mds.0.6084 boot_start 2:
replaying mds log
2019-08-20 09:18:19.240 7f6c6d95e700 2 mds.0.6084 boot_start 2: waiting
for purge queue recovered
2019-08-20 09:18:19.252 7f6c6e960700 4 mds.0.purge_queue operator():
open complete
2019-08-20 09:18:19.252 7f6c6d95e700 1 mds.0.6084 Finished replaying
journal
2019-08-20 09:18:19.252 7f6c6d95e700 1 mds.0.6084 making mds journal
writeable
2019-08-20 09:18:19.252 7f6c6d95e700 2 mds.0.6084 i am not alone,
moving to state resolve
2019-08-20 09:18:19.252 7f6c6d95e700 3 mds.0.6084 request_state up:resolve
2019-08-20 09:18:19.252 7f6c6d95e700 5 mds.beacon.juju-d0f708-3-lxd-1
set_want_state: up:replay -> up:resolve
2019-08-20 09:18:19.252 7f6c6d95e700 5 mds.beacon.juju-d0f708-3-lxd-1
Sending beacon up:resolve seq 40
2019-08-20 09:18:19.784 7f6c74353700 1 mds.juju-d0f708-3-lxd-1 Updating
MDS map to version 6085 from mon.0
2019-08-20 09:18:19.784 7f6c74353700 1 mds.0.6084 handle_mds_map i am
now mds.0.6084
2019-08-20 09:18:19.784 7f6c74353700 1 mds.0.6084 handle_mds_map state
change up:replay --> up:resolve
2019-08-20 09:18:19.784 7f6c74353700 1 mds.0.6084 resolve_start
2019-08-20 09:18:19.784 7f6c74353700 1 mds.0.6084 reopen_log
2019-08-20 09:18:19.784 7f6c74353700 1 mds.0.6084 recovery set is 1
2019-08-20 09:18:19.784 7f6c76357700 5 mds.beacon.juju-d0f708-3-lxd-1
received beacon reply up:resolve seq 40 rtt 0.531996
2019-08-20 09:18:19.784 7f6c74353700 5 mds.juju-d0f708-3-lxd-1
handle_mds_map old map epoch 6085 <= 6085, discarding
2019-08-20 09:18:19.788 7f6c74353700 1 mds.0.6084 resolve_done
2019-08-20 09:18:19.788 7f6c74353700 3 mds.0.6084 request_state
up:reconnect
2019-08-20 09:18:19.788 7f6c74353700 5 mds.beacon.juju-d0f708-3-lxd-1
set_want_state: up:resolve -> up:reconnect
2019-08-20 09:18:19.788 7f6c74353700 5 mds.beacon.juju-d0f708-3-lxd-1
Sending beacon up:reconnect seq 41
2019-08-20 09:18:20.804 7f6c74353700 1 mds.juju-d0f708-3-lxd-1 Updating
MDS map to version 6086 from mon.0
2019-08-20 09:18:20.804 7f6c74353700 1 mds.0.6084 handle_mds_map i am
now mds.0.6084
2019-08-20 09:18:20.804 7f6c74353700 1 mds.0.6084 handle_mds_map state
change up:resolve --> up:reconnect
2019-08-20 09:18:20.804 7f6c74353700 1 mds.0.6084 reconnect_start
2019-08-20 09:18:20.804 7f6c74353700 4 mds.0.6084 reconnect_start:
killed 0 blacklisted sessions (38 blacklist entries, 70)
2019-08-20 09:18:20.804 7f6c74353700 1 mds.0.server reconnect_clients
-- 70 sessions
2019-08-20 09:18:20.804 7f6c76357700 5 mds.beacon.juju-d0f708-3-lxd-1
received beacon reply up:reconnect seq 41 rtt 1.01599
2019-08-20 09:18:20.804 7f6c74353700 3 mds.0.server not active yet, waiting
2019-08-20 09:18:20.804 7f6c74353700 0 log_channel(cluster) log [DBG] :
reconnect by client.89616 130.149.2.137:0/3205297448 after 0
2019-08-20 09:18:20.804 7f6c74353700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:20.812 7f6c74353700 0 log_channel(cluster) log [DBG] :
reconnect by client.12124 172.28.9.23:0/346444993 after 0.00799994
2019-08-20 09:18:20.812 7f6c74353700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:20.812 7f6c74353700 3 mds.0.server not active yet, waiting
2019-08-20 09:18:20.812 7f6c74353700 0 log_channel(cluster) log [DBG] :
reconnect by client.48511 172.28.9.30:0/3829735889 after 0.00799994
2019-08-20 09:18:20.812 7f6c74353700 0 log_channel(cluster) do_log log
to syslog
[… more of these …]
2019-08-20 09:18:20.832 7f6c74353700 3 mds.0.server not active yet, waiting
2019-08-20 09:18:20.832 7f6c74353700 0 log_channel(cluster) log [DBG] :
reconnect by client.13108 10.175.4.19:0/375032773 after 0.0279998
2019-08-20 09:18:20.832 7f6c74353700 0 log_channel(cluster) do_log log
to syslog
[… more of these …]
2019-08-20 09:18:20.836 7f6c74353700 3 mds.0.server not active yet, waiting
2019-08-20 09:18:20.836 7f6c74353700 3 mds.0.server not active yet, waiting
2019-08-20 09:18:20.836 7f6c74353700 0 log_channel(cluster) log [DBG] :
reconnect by client.12139 10.175.4.16:0/1904855023 after 0.0319998
2019-08-20 09:18:20.836 7f6c74353700 0 log_channel(cluster) do_log log
to syslog
[… more of these …]
2019-08-20 09:18:20.896 7f6c74353700 1 mds.0.6084 reconnect_done
2019-08-20 09:18:20.896 7f6c74353700 3 mds.0.6084 request_state up:rejoin
2019-08-20 09:18:20.896 7f6c74353700 5 mds.beacon.juju-d0f708-3-lxd-1
set_want_state: up:reconnect -> up:rejoin
2019-08-20 09:18:20.896 7f6c74353700 5 mds.beacon.juju-d0f708-3-lxd-1
Sending beacon up:rejoin seq 42
2019-08-20 09:18:21.880 7f6c74353700 1 mds.juju-d0f708-3-lxd-1 Updating
MDS map to version 6087 from mon.0
2019-08-20 09:18:21.880 7f6c74353700 1 mds.0.6084 handle_mds_map i am
now mds.0.6084
2019-08-20 09:18:21.880 7f6c74353700 1 mds.0.6084 handle_mds_map state
change up:reconnect --> up:rejoin
2019-08-20 09:18:21.880 7f6c74353700 1 mds.0.6084 rejoin_start
2019-08-20 09:18:21.880 7f6c74353700 1 mds.0.6084 rejoin_joint_start
2019-08-20 09:18:21.880 7f6c74353700 5 mds.juju-d0f708-3-lxd-1
handle_mds_map old map epoch 6087 <= 6087, discarding
2019-08-20 09:18:21.880 7f6c76357700 5 mds.beacon.juju-d0f708-3-lxd-1
received beacon reply up:rejoin seq 42 rtt 0.983993
2019-08-20 09:18:21.888 7f6c6c95c700 5 mds.0.log _submit_thread
109124040146~3684 : ESessions 70 opens cmapv 968060
2019-08-20 09:18:21.956 7f6c74353700 5 mds.0.cache open_snaprealms has
unconnected snaprealm:
2019-08-20 09:18:21.956 7f6c74353700 5 mds.0.cache 0x10000000045
{client.79271/21}
2019-08-20 09:18:21.956 7f6c74353700 5 mds.0.cache 0x10000003f05
{client.79283/27}
2019-08-20 09:18:21.956 7f6c74353700 1 mds.0.6084 rejoin_done
2019-08-20 09:18:21.956 7f6c74353700 3 mds.0.6084 request_state up:active
2019-08-20 09:18:21.956 7f6c74353700 5 mds.beacon.juju-d0f708-3-lxd-1
set_want_state: up:rejoin -> up:active
2019-08-20 09:18:21.956 7f6c74353700 5 mds.beacon.juju-d0f708-3-lxd-1
Sending beacon up:active seq 43
2019-08-20 09:18:22.516 7f6c74353700 3 mds.0.server
handle_client_session client_session(request_renewcaps seq 77763) from
client.12127
2019-08-20 09:18:22.916 7f6c74353700 1 mds.juju-d0f708-3-lxd-1 Updating
MDS map to version 6088 from mon.0
2019-08-20 09:18:22.916 7f6c74353700 1 mds.0.6084 handle_mds_map i am
now mds.0.6084
2019-08-20 09:18:22.916 7f6c74353700 1 mds.0.6084 handle_mds_map state
change up:rejoin --> up:active
2019-08-20 09:18:22.916 7f6c74353700 1 mds.0.6084 recovery_done --
successful recovery!
2019-08-20 09:18:22.916 7f6c74353700 1 mds.0.6084 active_start
2019-08-20 09:18:22.916 7f6c76357700 5 mds.beacon.juju-d0f708-3-lxd-1
received beacon reply up:active seq 43 rtt 0.959993
2019-08-20 09:18:22.916 7f6c74353700 4 mds.0.6084
set_osd_epoch_barrier: epoch=7444
2019-08-20 09:18:22.920 7f6c74353700 4 mds.0.server
handle_client_request client_request(client.89616:241 lssnap
#0x10000000000 2019-08-20 09:17:49.274589 RETRY=1 caller_uid=0,
caller_gid=0{}) v2
2019-08-20 09:18:22.920 7f6c74353700 5 mds.0.server waiting for root
2019-08-20 09:18:22.920 7f6c74353700 4 mds.0.server
handle_client_request client_request(client.48511:30590 getattr
pAsLsXsFs #0x10000007b25 2019-08-20 09:17:54.991938 RETRY=1
caller_uid=0, caller_gid=0{}) v2
2019-08-20 09:18:22.920 7f6c74353700 5 mds.0.server waiting for root
2019-08-20 09:18:22.920 7f6c74353700 4 mds.0.server
handle_client_request client_request(client.13108:3485788 getattr Fs
#0x10000002b59 2019-08-20 09:17:47.771282 RETRY=1 caller_uid=1000,
caller_gid=1000{}) v2
2019-08-20 09:18:22.920 7f6c74353700 5 mds.0.server waiting for root
2019-08-20 09:18:22.920 7f6c74353700 4 mds.0.server
handle_client_request client_request(client.12139:3732538 getattr
pAsLsXsFs #0x10000002b59 2019-08-20 09:17:47.563937 RETRY=1
caller_uid=0, caller_gid=0{}) v2
[etc. …]
ceph.log on a mon:
2019-08-20 09:16:42.867 7f5a76512700 0 log_channel(cluster) log [DBG] :
fsmap ceph-fs-2/2/2 up
{0=juju-d0f708-9-lxd-1=up:active,1=juju-d0f708-10-lxd-1=up:active}, 2
up:standby-replay
2019-08-20 09:16:42.867 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:17:25.599 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6083 new map
2019-08-20 09:17:25.599 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6083 print_map
e6083
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'ceph-fs' (1)
fs_name ceph-fs
epoch 6083
flags 12
created 2019-08-05 12:21:23.208718
modified 2019-08-20 09:17:25.565361
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 7444
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=89667,1=89673}
failed
damaged
stopped
data_pools [2,3,4,5]
metadata_pool 1
inline_data disabled
balancer
standby_count_wanted 1
89667: 172.28.9.20:6800/2237168008 'juju-d0f708-9-lxd-1' mds.0.6070
up:active seq 28 export_targets=1
89679: 172.28.9.19:6800/1509831355 'juju-d0f708-3-lxd-1' mds.0.0
up:standby-replay seq 1
89673: 172.28.9.21:6800/2008779502 'juju-d0f708-10-lxd-1' mds.1.6073
up:active seq 5 export_targets=0
89685: 172.28.9.18:6800/1458048941 'juju-d0f708-5-lxd-1' mds.1.0
up:standby-replay seq 1
2019-08-20 09:17:25.599 7f5a76512700 0 log_channel(cluster) log [DBG] :
fsmap ceph-fs-2/2/2 up
{0=juju-d0f708-9-lxd-1=up:active,1=juju-d0f708-10-lxd-1=up:active}, 2
up:standby-replay
2019-08-20 09:17:25.599 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:06.518 7f5a80580700 0 log_channel(audit) log [DBG] :
from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch
2019-08-20 09:18:06.518 7f5a80580700 0 log_channel(audit) do_log log to
syslog
2019-08-20 09:18:06.518 7f5a80580700 0 log_channel(audit) log [DBG] :
from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished
2019-08-20 09:18:06.518 7f5a80580700 0 log_channel(audit) do_log log to
syslog
2019-08-20 09:18:18.734 7f5a7cd1f700 0 log_channel(cluster) log [WRN] :
daemon mds.juju-d0f708-9-lxd-1 is not responding, replacing it as rank 0
with standby daemon mds.juju-d0f708-3-lxd-1
2019-08-20 09:18:18.734 7f5a7cd1f700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:18.738 7f5a7cd1f700 0 log_channel(cluster) log [WRN] :
Health check failed: 1 filesystem is degraded (FS_DEGRADED)
2019-08-20 09:18:18.738 7f5a7cd1f700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:18.758 7f5a76512700 0 log_channel(cluster) log [DBG] :
osdmap e7445: 389 total, 389 up, 389 in
2019-08-20 09:18:18.758 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:18.774 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6084 new map
2019-08-20 09:18:18.774 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6084 print_map
e6084
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'ceph-fs' (1)
fs_name ceph-fs
epoch 6084
flags 12
created 2019-08-05 12:21:23.208718
modified 2019-08-20 09:18:18.742755
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 7445
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=89679,1=89673}
failed
damaged
stopped
data_pools [2,3,4,5]
metadata_pool 1
inline_data disabled
balancer
standby_count_wanted 1
89679: 172.28.9.19:6800/1509831355 'juju-d0f708-3-lxd-1' mds.0.6084
up:replay seq 1
89673: 172.28.9.21:6800/2008779502 'juju-d0f708-10-lxd-1' mds.1.6073
up:active seq 5 export_targets=0
89685: 172.28.9.18:6800/1458048941 'juju-d0f708-5-lxd-1' mds.1.0
up:standby-replay seq 1
2019-08-20 09:18:18.774 7f5a76512700 0 log_channel(cluster) log [DBG] :
fsmap ceph-fs-2/2/2 up
{0=juju-d0f708-3-lxd-1=up:replay,1=juju-d0f708-10-lxd-1=up:active}, 1
up:standby-replay
2019-08-20 09:18:18.774 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:19.782 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6085 new map
2019-08-20 09:18:19.782 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6085 print_map
e6085
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'ceph-fs' (1)
fs_name ceph-fs
epoch 6085
flags 12
created 2019-08-05 12:21:23.208718
modified 2019-08-20 09:18:19.776171
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 7445
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=89679,1=89673}
failed
damaged
stopped
data_pools [2,3,4,5]
metadata_pool 1
inline_data disabled
balancer
standby_count_wanted 1
89679: 172.28.9.19:6800/1509831355 'juju-d0f708-3-lxd-1' mds.0.6084
up:resolve seq 40
89673: 172.28.9.21:6800/2008779502 'juju-d0f708-10-lxd-1' mds.1.6073
up:active seq 5 export_targets=0
89685: 172.28.9.18:6800/1458048941 'juju-d0f708-5-lxd-1' mds.1.0
up:standby-replay seq 1
2019-08-20 09:18:19.782 7f5a76512700 0 log_channel(cluster) log [DBG] :
mds.0 172.28.9.19:6800/1509831355 up:resolve
2019-08-20 09:18:19.782 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:19.782 7f5a76512700 0 log_channel(cluster) log [DBG] :
fsmap ceph-fs-2/2/2 up
{0=juju-d0f708-3-lxd-1=up:resolve,1=juju-d0f708-10-lxd-1=up:active}, 1
up:standby-replay
2019-08-20 09:18:19.782 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:20.802 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6086 new map
2019-08-20 09:18:20.802 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6086 print_map
e6086
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'ceph-fs' (1)
fs_name ceph-fs
epoch 6086
flags 12
created 2019-08-05 12:21:23.208718
modified 2019-08-20 09:18:20.785775
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 7445
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=89679,1=89673}
failed
damaged
stopped
data_pools [2,3,4,5]
metadata_pool 1
inline_data disabled
balancer
standby_count_wanted 1
89679: 172.28.9.19:6800/1509831355 'juju-d0f708-3-lxd-1' mds.0.6084
up:reconnect seq 41
89673: 172.28.9.21:6800/2008779502 'juju-d0f708-10-lxd-1' mds.1.6073
up:active seq 5 export_targets=0
89685: 172.28.9.18:6800/1458048941 'juju-d0f708-5-lxd-1' mds.1.0
up:standby-replay seq 1
2019-08-20 09:18:20.802 7f5a76512700 0 log_channel(cluster) log [DBG] :
mds.0 172.28.9.19:6800/1509831355 up:reconnect
2019-08-20 09:18:20.802 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:20.802 7f5a76512700 0 log_channel(cluster) log [DBG] :
fsmap ceph-fs-2/2/2 up
{0=juju-d0f708-3-lxd-1=up:reconnect,1=juju-d0f708-10-lxd-1=up:active}, 1
up:standby-replay
2019-08-20 09:18:20.802 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:21.834 7f5a7cd1f700 0 log_channel(cluster) log [WRN] :
Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST)
2019-08-20 09:18:21.834 7f5a7cd1f700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:21.878 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6087 new map
2019-08-20 09:18:21.878 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6087 print_map
e6087
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'ceph-fs' (1)
fs_name ceph-fs
epoch 6087
flags 12
created 2019-08-05 12:21:23.208718
modified 2019-08-20 09:18:21.837992
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 7445
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=89679,1=89673}
failed
damaged
stopped
data_pools [2,3,4,5]
metadata_pool 1
inline_data disabled
balancer
standby_count_wanted 1
89679: 172.28.9.19:6800/1509831355 'juju-d0f708-3-lxd-1' mds.0.6084
up:rejoin seq 42
89673: 172.28.9.21:6800/2008779502 'juju-d0f708-10-lxd-1' mds.1.6073
up:active seq 45 export_targets=0
89685: 172.28.9.18:6800/1458048941 'juju-d0f708-5-lxd-1' mds.1.0
up:standby-replay seq 1
2019-08-20 09:18:21.882 7f5a76512700 0 log_channel(cluster) log [DBG] :
mds.1 172.28.9.21:6800/2008779502 up:active
2019-08-20 09:18:21.882 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:21.882 7f5a76512700 0 log_channel(cluster) log [DBG] :
mds.0 172.28.9.19:6800/1509831355 up:rejoin
2019-08-20 09:18:21.882 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:21.882 7f5a76512700 0 log_channel(cluster) log [DBG] :
fsmap ceph-fs-2/2/2 up
{0=juju-d0f708-3-lxd-1=up:rejoin,1=juju-d0f708-10-lxd-1=up:active}, 1
up:standby-replay
2019-08-20 09:18:21.882 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:21.958 7f5a7a51a700 0 log_channel(cluster) log [INF] :
daemon mds.juju-d0f708-3-lxd-1 is now active in filesystem ceph-fs as rank 0
2019-08-20 09:18:21.958 7f5a7a51a700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:22.878 7f5a7cd1f700 0 log_channel(cluster) log [INF] :
Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded)
2019-08-20 09:18:22.878 7f5a7cd1f700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:22.914 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6088 new map
2019-08-20 09:18:22.914 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6088 print_map
e6088
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'ceph-fs' (1)
fs_name ceph-fs
epoch 6088
flags 12
created 2019-08-05 12:21:23.208718
modified 2019-08-20 09:18:22.882183
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 7445
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=89679,1=89673}
failed
damaged
stopped
data_pools [2,3,4,5]
metadata_pool 1
inline_data disabled
balancer
standby_count_wanted 1
89679: 172.28.9.19:6800/1509831355 'juju-d0f708-3-lxd-1' mds.0.6084
up:active seq 43
89673: 172.28.9.21:6800/2008779502 'juju-d0f708-10-lxd-1' mds.1.6073
up:active seq 45 export_targets=0
89685: 172.28.9.18:6800/1458048941 'juju-d0f708-5-lxd-1' mds.1.0
up:standby-replay seq 1
2019-08-20 09:18:22.914 7f5a76512700 0 log_channel(cluster) log [DBG] :
mds.0 172.28.9.19:6800/1509831355 up:active
2019-08-20 09:18:22.914 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:22.914 7f5a76512700 0 log_channel(cluster) log [DBG] :
fsmap ceph-fs-2/2/2 up
{0=juju-d0f708-3-lxd-1=up:active,1=juju-d0f708-10-lxd-1=up:active}, 1
up:standby-replay
2019-08-20 09:18:22.914 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:23.646 7f5a7a51a700 0 mon.ceph-mon-01@0(leader) e2
handle_command mon_command({"prefix": "mds metadata", "who":
"juju-d0f708-9-lxd-1"} v 0) v1
2019-08-20 09:18:23.646 7f5a7a51a700 0 log_channel(audit) log [DBG] :
from='mgr.86070 172.28.9.11:0/1868533' entity='mgr.ceph-mon-01'
cmd=[{"prefix": "mds metadata", "who": "juju-d0f708-9-lxd-1"}]: dispatch
2019-08-20 09:18:23.646 7f5a7a51a700 0 log_channel(audit) do_log log to
syslog
2019-08-20 09:18:53.745 7f5a7cd1f700 0 log_channel(cluster) log [WRN] :
daemon mds.juju-d0f708-10-lxd-1 is not responding, replacing it as rank
1 with standby daemon mds.juju-d0f708-5-lxd-1
2019-08-20 09:18:53.745 7f5a7cd1f700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:53.773 7f5a7cd1f700 0 log_channel(cluster) log [WRN] :
Health check failed: 1 filesystem is degraded (FS_DEGRADED)
2019-08-20 09:18:53.773 7f5a7cd1f700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:53.773 7f5a7cd1f700 0 log_channel(cluster) log [WRN] :
Health check failed: insufficient standby MDS daemons available
(MDS_INSUFFICIENT_STANDBY)
2019-08-20 09:18:53.773 7f5a7cd1f700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:53.773 7f5a7cd1f700 0 log_channel(cluster) log [INF] :
Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests)
2019-08-20 09:18:53.773 7f5a7cd1f700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:53.805 7f5a76512700 0 log_channel(cluster) log [DBG] :
osdmap e7446: 389 total, 389 up, 389 in
2019-08-20 09:18:53.805 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:53.817 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6089 new map
2019-08-20 09:18:53.817 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6089 print_map
e6089
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'ceph-fs' (1)
fs_name ceph-fs
epoch 6089
flags 12
created 2019-08-05 12:21:23.208718
modified 2019-08-20 09:18:53.778337
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 7446
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=89679,1=89685}
failed
damaged
stopped
data_pools [2,3,4,5]
metadata_pool 1
inline_data disabled
balancer
standby_count_wanted 1
89679: 172.28.9.19:6800/1509831355 'juju-d0f708-3-lxd-1' mds.0.6084
up:active seq 43
89685: 172.28.9.18:6800/1458048941 'juju-d0f708-5-lxd-1' mds.1.6089
up:replay seq 1
2019-08-20 09:18:53.821 7f5a76512700 0 log_channel(cluster) log [DBG] :
fsmap ceph-fs-2/2/2 up
{0=juju-d0f708-3-lxd-1=up:active,1=juju-d0f708-5-lxd-1=up:replay}
2019-08-20 09:18:53.821 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:54.865 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6090 new map
2019-08-20 09:18:54.865 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6090 print_map
e6090
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'ceph-fs' (1)
fs_name ceph-fs
epoch 6090
flags 12
created 2019-08-05 12:21:23.208718
modified 2019-08-20 09:18:54.823816
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 7446
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=89679,1=89685}
failed
damaged
stopped
data_pools [2,3,4,5]
metadata_pool 1
inline_data disabled
balancer
standby_count_wanted 1
89679: 172.28.9.19:6800/1509831355 'juju-d0f708-3-lxd-1' mds.0.6084
up:active seq 43
89685: 172.28.9.18:6800/1458048941 'juju-d0f708-5-lxd-1' mds.1.6089
up:resolve seq 34
2019-08-20 09:18:54.865 7f5a76512700 0 log_channel(cluster) log [DBG] :
mds.1 172.28.9.18:6800/1458048941 up:resolve
2019-08-20 09:18:54.865 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:54.865 7f5a76512700 0 log_channel(cluster) log [DBG] :
fsmap ceph-fs-2/2/2 up
{0=juju-d0f708-3-lxd-1=up:active,1=juju-d0f708-5-lxd-1=up:resolve}
2019-08-20 09:18:54.865 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:55.781 7f5a7a51a700 0 mon.ceph-mon-01@0(leader) e2
handle_command mon_command({"prefix": "mds metadata", "who":
"juju-d0f708-10-lxd-1"} v 0) v1
2019-08-20 09:18:55.781 7f5a7a51a700 0 log_channel(audit) log [DBG] :
from='mgr.86070 172.28.9.11:0/1868533' entity='mgr.ceph-mon-01'
cmd=[{"prefix": "mds metadata", "who": "juju-d0f708-10-lxd-1"}]: dispatch
2019-08-20 09:18:55.781 7f5a7a51a700 0 log_channel(audit) do_log log to
syslog
2019-08-20 09:18:55.901 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6091 new map
2019-08-20 09:18:55.901 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6091 print_map
e6091
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'ceph-fs' (1)
fs_name ceph-fs
epoch 6091
flags 12
created 2019-08-05 12:21:23.208718
modified 2019-08-20 09:18:55.868990
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 7446
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=89679,1=89685}
failed
damaged
stopped
data_pools [2,3,4,5]
metadata_pool 1
inline_data disabled
balancer
standby_count_wanted 1
89679: 172.28.9.19:6800/1509831355 'juju-d0f708-3-lxd-1' mds.0.6084
up:active seq 43
89685: 172.28.9.18:6800/1458048941 'juju-d0f708-5-lxd-1' mds.1.6089
up:reconnect seq 35
2019-08-20 09:18:55.901 7f5a76512700 0 log_channel(cluster) log [DBG] :
mds.1 172.28.9.18:6800/1458048941 up:reconnect
2019-08-20 09:18:55.901 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:55.901 7f5a76512700 0 log_channel(cluster) log [DBG] :
fsmap ceph-fs-2/2/2 up
{0=juju-d0f708-3-lxd-1=up:active,1=juju-d0f708-5-lxd-1=up:reconnect}
2019-08-20 09:18:55.901 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:56.977 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6092 new map
2019-08-20 09:18:56.977 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6092 print_map
e6092
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'ceph-fs' (1)
fs_name ceph-fs
epoch 6092
flags 12
created 2019-08-05 12:21:23.208718
modified 2019-08-20 09:18:56.937720
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 7446
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=89679,1=89685}
failed
damaged
stopped
data_pools [2,3,4,5]
metadata_pool 1
inline_data disabled
balancer
standby_count_wanted 1
89679: 172.28.9.19:6800/1509831355 'juju-d0f708-3-lxd-1' mds.0.6084
up:active seq 43
89685: 172.28.9.18:6800/1458048941 'juju-d0f708-5-lxd-1' mds.1.6089
up:rejoin seq 36
2019-08-20 09:18:56.977 7f5a76512700 0 log_channel(cluster) log [DBG] :
mds.1 172.28.9.18:6800/1458048941 up:rejoin
2019-08-20 09:18:56.977 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:56.977 7f5a76512700 0 log_channel(cluster) log [DBG] :
fsmap ceph-fs-2/2/2 up
{0=juju-d0f708-3-lxd-1=up:active,1=juju-d0f708-5-lxd-1=up:rejoin}
2019-08-20 09:18:56.977 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:57.077 7f5a7a51a700 0 log_channel(cluster) log [INF] :
daemon mds.juju-d0f708-5-lxd-1 is now active in filesystem ceph-fs as rank 1
2019-08-20 09:18:57.077 7f5a7a51a700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:57.993 7f5a7cd1f700 0 log_channel(cluster) log [WRN] :
Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST)
2019-08-20 09:18:57.993 7f5a7cd1f700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:57.993 7f5a7cd1f700 0 log_channel(cluster) log [INF] :
Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded)
2019-08-20 09:18:57.993 7f5a7cd1f700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:58.037 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6093 new map
2019-08-20 09:18:58.037 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6093 print_map
e6093
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'ceph-fs' (1)
fs_name ceph-fs
epoch 6093
flags 12
created 2019-08-05 12:21:23.208718
modified 2019-08-20 09:18:57.998584
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 7446
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=89679,1=89685}
failed
damaged
stopped
data_pools [2,3,4,5]
metadata_pool 1
inline_data disabled
balancer
standby_count_wanted 1
89679: 172.28.9.19:6800/1509831355 'juju-d0f708-3-lxd-1' mds.0.6084
up:active seq 52
89685: 172.28.9.18:6800/1458048941 'juju-d0f708-5-lxd-1' mds.1.6089
up:active seq 37
2019-08-20 09:18:58.037 7f5a76512700 0 log_channel(cluster) log [DBG] :
mds.1 172.28.9.18:6800/1458048941 up:active
2019-08-20 09:18:58.037 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:58.037 7f5a76512700 0 log_channel(cluster) log [DBG] :
mds.0 172.28.9.19:6800/1509831355 up:active
2019-08-20 09:18:58.037 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:18:58.037 7f5a76512700 0 log_channel(cluster) log [DBG] :
fsmap ceph-fs-2/2/2 up
{0=juju-d0f708-3-lxd-1=up:active,1=juju-d0f708-5-lxd-1=up:active}
2019-08-20 09:18:58.037 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:19:01.957 7f5a7a51a700 0 log_channel(cluster) log [INF] :
MDS health message cleared (mds.0): 4 slow requests are blocked > 30 secs
2019-08-20 09:19:01.957 7f5a7a51a700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:19:02.293 7f5a7cd1f700 0 log_channel(cluster) log [INF] :
Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests)
2019-08-20 09:19:02.293 7f5a7cd1f700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:19:02.329 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6094 new map
2019-08-20 09:19:02.329 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6094 print_map
e6094
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'ceph-fs' (1)
fs_name ceph-fs
epoch 6094
flags 12
created 2019-08-05 12:21:23.208718
modified 2019-08-20 09:19:02.297056
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 7446
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=89679,1=89685}
failed
damaged
stopped
data_pools [2,3,4,5]
metadata_pool 1
inline_data disabled
balancer
standby_count_wanted 1
89679: 172.28.9.19:6800/1509831355 'juju-d0f708-3-lxd-1' mds.0.6084
up:active seq 53
89685: 172.28.9.18:6800/1458048941 'juju-d0f708-5-lxd-1' mds.1.6089
up:active seq 37
2019-08-20 09:19:02.329 7f5a76512700 0 log_channel(cluster) log [DBG] :
mds.0 172.28.9.19:6800/1509831355 up:active
2019-08-20 09:19:02.329 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:19:02.329 7f5a76512700 0 log_channel(cluster) log [DBG] :
fsmap ceph-fs-2/2/2 up
{0=juju-d0f708-3-lxd-1=up:active,1=juju-d0f708-5-lxd-1=up:active}
2019-08-20 09:19:02.329 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:19:07.013 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6095 new map
2019-08-20 09:19:07.013 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6095 print_map
e6095
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'ceph-fs' (1)
fs_name ceph-fs
epoch 6095
flags 12
created 2019-08-05 12:21:23.208718
modified 2019-08-20 09:19:06.975109
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 7446
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=89679,1=89685}
failed
damaged
stopped
data_pools [2,3,4,5]
metadata_pool 1
inline_data disabled
balancer
standby_count_wanted 1
89679: 172.28.9.19:6800/1509831355 'juju-d0f708-3-lxd-1' mds.0.6084
up:active seq 53
89685: 172.28.9.18:6800/1458048941 'juju-d0f708-5-lxd-1' mds.1.6089
up:active seq 37 export_targets=0
2019-08-20 09:19:07.013 7f5a76512700 0 log_channel(cluster) log [DBG] :
fsmap ceph-fs-2/2/2 up
{0=juju-d0f708-3-lxd-1=up:active,1=juju-d0f708-5-lxd-1=up:active}
2019-08-20 09:19:07.013 7f5a76512700 0 log_channel(cluster) do_log log
to syslog
2019-08-20 09:19:35.696 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6096 new map
2019-08-20 09:19:35.696 7f5a76512700 0 mon.ceph-mon-01@0(leader).mds
e6096 print_map
e6096
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
Filesystem 'ceph-fs' (1)
fs_name ceph-fs
epoch 6096
flags 12
created 2019-08-05 12:21:23.208718
modified 2019-08-20 09:19:35.657067
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
min_compat_client -1 (unspecified)
last_failure 0
last_failure_osd_epoch 7446
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in 0,1
up {0=89679,1=89685}
failed
damaged
stopped
data_pools [2,3,4,5]
metadata_pool 1
inline_data disabled
balancer
standby_count_wanted 1
89679: 172.28.9.19:6800/1509831355 'juju-d0f708-3-lxd-1' mds.0.6084
up:active seq 53 export_targets=1
89685: 172.28.9.18:6800/1458048941 'juju-d0f708-5-lxd-1' mds.1.6089
up:active seq 37 export_targets=0
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
