Hi,
Last week out MDSs started failing one after another, and could not be started anymore. After a lot of tinkering I found out that MDSs crashed after trying to rejoin the Cluster. The only Solution I found that, let them start again was resetting the journal vie cephfs-journal-tool. Now I have broken files all over the Cluster.
Before the crash the OSDs blocked tens of thousands of slow requests.
Can I somehow restore the broken files (I still have a backup of the journal) and how can I make sure that this doesn't happen agian. I am still not sure why this even happened.
This happened on ceph version 12.2.3.
This is the log of one MDS:
-224> 2018-03-22 15:52:47.310437 7fd5798fd700 1 -- x.x.1.17:6803/122963511 <== mon.0 x.x.1.17:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3611581813 0 0) 0x555883df2780 con 0x555883eb5000
-223> 2018-03-22 15:52:47.310482 7fd5798fd700 10 monclient(hunting): my global_id is 745317
-222> 2018-03-22 15:52:47.310634 7fd5798fd700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- 0x555883df2f00 con 0
-221> 2018-03-22 15:52:47.311096 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6789/0 conn(0x555883eb5000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=793748 cs=1 l=1). rx mon.0 seq 3 0x555883df2f00 auth_reply(proto 2 0 (0) Success) v1
-220> 2018-03-22 15:52:47.311178 7fd5798fd700 1 -- x.x.1.17:6803/122963511 <== mon.0 x.x.1.17:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 222+0+0 (1789869469 0 0) 0x555883df2f00 con 0x555883eb5000
-219> 2018-03-22 15:52:47.311319 7fd5798fd700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- 0x555883df2780 con 0
-218> 2018-03-22 15:52:47.312122 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6789/0 conn(0x555883eb5000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=793748 cs=1 l=1). rx mon.0 seq 4 0x555883df2780 auth_reply(proto 2 0 (0) Success) v1
-217> 2018-03-22 15:52:47.312208 7fd5798fd700 1 -- x.x.1.17:6803/122963511 <== mon.0 x.x.1.17:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 799+0+0 (4156877078 0 0) 0x555883df2780 con 0x555883eb5000
-216> 2018-03-22 15:52:47.312393 7fd5798fd700 1 monclient: found mon.filer1
-215> 2018-03-22 15:52:47.312416 7fd5798fd700 10 monclient: _send_mon_message to mon.filer1 at x.x.1.17:6789/0
-214> 2018-03-22 15:52:47.312427 7fd5798fd700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6789/0 -- mon_subscribe({monmap=0+}) v2 -- 0x555883c8ed80 con 0
-213> 2018-03-22 15:52:47.312461 7fd5798fd700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2018-03-22 15:52:17.312460)
-212> 2018-03-22 15:52:47.312477 7fd5798fd700 10 monclient: _send_mon_message to mon.filer1 at x.x.1.17:6789/0
-211> 2018-03-22 15:52:47.312482 7fd5798fd700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x555883df2f00 con 0
-210> 2018-03-22 15:52:47.312552 7fd580637200 5 monclient: authenticate success, global_id 745317
-209> 2018-03-22 15:52:47.312570 7fd580637200 10 monclient: wait_auth_rotating waiting (until 2018-03-22 15:53:17.312568)
-208> 2018-03-22 15:52:47.312776 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6789/0 conn(0x555883eb5000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=793748 cs=1 l=1). rx mon.0 seq 5 0x555883c8f8c0 mon_map magic: 0 v1
-207> 2018-03-22 15:52:47.312841 7fd5798fd700 1 -- x.x.1.17:6803/122963511 <== mon.0 x.x.1.17:6789/0 5 ==== mon_map magic: 0 v1 ==== 433+0+0 (493202164 0 0) 0x555883c8f8c0 con 0x555883eb5000
-206> 2018-03-22 15:52:47.312868 7fd5798fd700 10 monclient: handle_monmap mon_map magic: 0 v1
-205> 2018-03-22 15:52:47.312892 7fd5798fd700 10 monclient: got monmap 7, mon.filer1 is now rank 0
-204> 2018-03-22 15:52:47.312901 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6789/0 conn(0x555883eb5000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=793748 cs=1 l=1). rx mon.0 seq 6 0x555883df2f00 auth_reply(proto 2 0 (0) Success) v1
-203> 2018-03-22 15:52:47.312900 7fd5798fd700 10 monclient: dump:
epoch 7
fsid a5473adc-cfb8-4672-883e-40f5f6541a36
last_changed 2017-12-08 10:38:51.267030
created 2017-01-20 17:05:29.092109
0: x.x.1.17:6789/0 mon.filer1
1: x.x.1.18:6789/0 mon.filer2
2: x.x.1.21:6789/0 mon.master1
-202> 2018-03-22 15:52:47.312950 7fd5798fd700 1 -- x.x.1.17:6803/122963511 <== mon.0 x.x.1.17:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (1424514407 0 0) 0x555883df2f00 con 0x555883eb5000
-201> 2018-03-22 15:52:47.313072 7fd5798fd700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-03-22 15:52:17.313071)
-200> 2018-03-22 15:52:47.313108 7fd580637200 10 monclient: wait_auth_rotating done
-199> 2018-03-22 15:52:47.313172 7fd580637200 10 monclient: _renew_subs
-198> 2018-03-22 15:52:47.313179 7fd580637200 10 monclient: _send_mon_message to mon.filer1 at x.x.1.17:6789/0
-197> 2018-03-22 15:52:47.313188 7fd580637200 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6789/0 -- mon_subscribe({mdsmap=0+,mgrmap=0+}) v2 -- 0x555883c8efc0 con 0
-196> 2018-03-22 15:52:47.313209 7fd580637200 5 asok(0x555883c661e0) register_command status hook 0x555883c362f0
-195> 2018-03-22 15:52:47.313219 7fd580637200 5 asok(0x555883c661e0) register_command dump_ops_in_flight hook 0x555883c362f0
-194> 2018-03-22 15:52:47.313224 7fd580637200 5 asok(0x555883c661e0) register_command ops hook 0x555883c362f0
-193> 2018-03-22 15:52:47.313231 7fd580637200 5 asok(0x555883c661e0) register_command dump_blocked_ops hook 0x555883c362f0
-192> 2018-03-22 15:52:47.313234 7fd580637200 5 asok(0x555883c661e0) register_command dump_historic_ops hook 0x555883c362f0
-191> 2018-03-22 15:52:47.313238 7fd580637200 5 asok(0x555883c661e0) register_command dump_historic_ops_by_duration hook 0x555883c362f0
-190> 2018-03-22 15:52:47.313246 7fd580637200 5 asok(0x555883c661e0) register_command scrub_path hook 0x555883c362f0
-189> 2018-03-22 15:52:47.313250 7fd580637200 5 asok(0x555883c661e0) register_command tag path hook 0x555883c362f0
-188> 2018-03-22 15:52:47.313259 7fd580637200 5 asok(0x555883c661e0) register_command flush_path hook 0x555883c362f0
-187> 2018-03-22 15:52:47.313262 7fd580637200 5 asok(0x555883c661e0) register_command export dir hook 0x555883c362f0
-186> 2018-03-22 15:52:47.313268 7fd580637200 5 asok(0x555883c661e0) register_command dump cache hook 0x555883c362f0
-185> 2018-03-22 15:52:47.313275 7fd580637200 5 asok(0x555883c661e0) register_command cache status hook 0x555883c362f0
-184> 2018-03-22 15:52:47.313279 7fd580637200 5 asok(0x555883c661e0) register_command dump tree hook 0x555883c362f0
-183> 2018-03-22 15:52:47.313284 7fd580637200 5 asok(0x555883c661e0) register_command session evict hook 0x555883c362f0
-182> 2018-03-22 15:52:47.313291 7fd580637200 5 asok(0x555883c661e0) register_command osdmap barrier hook 0x555883c362f0
-181> 2018-03-22 15:52:47.313304 7fd580637200 5 asok(0x555883c661e0) register_command session ls hook 0x555883c362f0
-180> 2018-03-22 15:52:47.313315 7fd580637200 5 asok(0x555883c661e0) register_command flush journal hook 0x555883c362f0
-179> 2018-03-22 15:52:47.313323 7fd580637200 5 asok(0x555883c661e0) register_command force_readonly hook 0x555883c362f0
-178> 2018-03-22 15:52:47.313333 7fd580637200 5 asok(0x555883c661e0) register_command get subtrees hook 0x555883c362f0
-177> 2018-03-22 15:52:47.313346 7fd580637200 5 asok(0x555883c661e0) register_command dirfrag split hook 0x555883c362f0
-176> 2018-03-22 15:52:47.313361 7fd580637200 5 asok(0x555883c661e0) register_command dirfrag merge hook 0x555883c362f0
-175> 2018-03-22 15:52:47.313375 7fd580637200 5 asok(0x555883c661e0) register_command dirfrag ls hook 0x555883c362f0
-174> 2018-03-22 15:52:47.313528 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6789/0 conn(0x555883eb5000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=793748 cs=1 l=1). rx mon.0 seq 7 0x555883df3180 mdsmap(e 5323) v1
-173> 2018-03-22 15:52:47.313589 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6789/0 conn(0x555883eb5000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=793748 cs=1 l=1). rx mon.0 seq 8 0x555883c8a600 mgrmap(e 279) v1
-172> 2018-03-22 15:52:47.313582 7fd5798fd700 1 -- x.x.1.17:6803/122963511 <== mon.0 x.x.1.17:6789/0 7 ==== mdsmap(e 5323) v1 ==== 464+0+0 (1833395745 0 0) 0x555883df3180 con 0x555883eb5000
-171> 2018-03-22 15:52:47.313829 7fd580637200 10 monclient: _send_mon_message to mon.filer1 at x.x.1.17:6789/0
-170> 2018-03-22 15:52:47.313838 7fd580637200 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6789/0 -- mdsbeacon(745317/filer1 up:boot seq 1 v0) v7 -- 0x555883f20380 con 0
-169> 2018-03-22 15:52:47.313888 7fd5798fd700 5 mds.filer1 handle_mds_map epoch 5323 from mon.0
-168> 2018-03-22 15:52:47.313954 7fd5798fd700 1 -- x.x.1.17:6803/122963511 <== mon.0 x.x.1.17:6789/0 8 ==== mgrmap(e 279) v1 ==== 476+0+0 (79426338 0 0) 0x555883c8a600 con 0x555883eb5000
-167> 2018-03-22 15:52:47.313982 7fd5798fd700 4 mgrc handle_mgr_map Got map version 279
-166> 2018-03-22 15:52:47.313988 7fd5798fd700 4 mgrc handle_mgr_map Active mgr is now x.x.1.17:6800/63308
-165> 2018-03-22 15:52:47.313993 7fd5798fd700 4 mgrc reconnect Starting new session with x.x.1.17:6800/63308
-164> 2018-03-22 15:52:47.314044 7fd5798fd700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6800/63308 -- mgropen(unknown.filer1) v2 -- 0x555883df2780 con 0
-163> 2018-03-22 15:52:47.314639 7fd57b89e700 2 -- x.x.1.17:6803/122963511 >> x.x.1.17:6800/63308 conn(0x555883f2e800 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
-162> 2018-03-22 15:52:47.314984 7fd57b89e700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6800/63308 conn(0x555883f2e800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=251 cs=1 l=1). rx mgr.744273 seq 1 0x555883c66b40 mgrconfigure(period=5, threshold=5) v2
-161> 2018-03-22 15:52:47.315023 7fd5798fd700 1 -- x.x.1.17:6803/122963511 <== mgr.744273 x.x.1.17:6800/63308 1 ==== mgrconfigure(period=5, threshold=5) v2 ==== 8+0+0 (3460719617 0 0) 0x555883c66b40 con 0x555883f2e800
-160> 2018-03-22 15:52:47.315044 7fd5798fd700 4 mgrc handle_mgr_configure stats_period=5
-159> 2018-03-22 15:52:47.315047 7fd5798fd700 4 mgrc handle_mgr_configure updated stats threshold: 5
-158> 2018-03-22 15:52:47.315057 7fd5798fd700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6800/63308 -- mgrreport(unknown.filer1 +0-0 packed 6) v5 -- 0x555883c8a600 con 0
-157> 2018-03-22 15:52:51.313978 7fd5768f7700 10 monclient: _send_mon_message to mon.filer1 at x.x.1.17:6789/0
-156> 2018-03-22 15:52:51.313995 7fd5768f7700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6789/0 -- mdsbeacon(745317/filer1 up:boot seq 2 v5323) v7 -- 0x555883f20700 con 0
-155> 2018-03-22 15:52:51.967269 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6789/0 conn(0x555883eb5000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=793748 cs=1 l=1). rx mon.0 seq 9 0x555883df3400 mdsmap(e 5324) v1
-154> 2018-03-22 15:52:51.967345 7fd5798fd700 1 -- x.x.1.17:6803/122963511 <== mon.0 x.x.1.17:6789/0 9 ==== mdsmap(e 5324) v1 ==== 588+0+0 (543550856 0 0) 0x555883df3400 con 0x555883eb5000
-153> 2018-03-22 15:52:51.967377 7fd5798fd700 5 mds.filer1 handle_mds_map epoch 5324 from mon.0
-152> 2018-03-22 15:52:51.967463 7fd5798fd700 1 mds.filer1 handle_mds_map standby
-151> 2018-03-22 15:52:51.967741 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6789/0 conn(0x555883eb5000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=793748 cs=1 l=1). rx mon.0 seq 10 0x555883f20700 mdsbeacon(745317/filer1 up:boot seq 2 v5324) v7
-150> 2018-03-22 15:52:51.967790 7fd5798fd700 1 -- x.x.1.17:6803/122963511 <== mon.0 x.x.1.17:6789/0 10 ==== mdsbeacon(745317/filer1 up:boot seq 2 v5324) v7 ==== 139+0+0 (3023608128 0 0) 0x555883f20700 con 0x555883eb5000
-149> 2018-03-22 15:52:52.068410 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6789/0 conn(0x555883eb5000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=793748 cs=1 l=1). rx mon.0 seq 11 0x555883df3680 mdsmap(e 5325) v1
-148> 2018-03-22 15:52:52.068475 7fd5798fd700 1 -- x.x.1.17:6803/122963511 <== mon.0 x.x.1.17:6789/0 11 ==== mdsmap(e 5325) v1 ==== 625+0+0 (3633367692 0 0) 0x555883df3680 con 0x555883eb5000
-147> 2018-03-22 15:52:52.068506 7fd5798fd700 5 mds.filer1 handle_mds_map epoch 5325 from mon.0
-146> 2018-03-22 15:52:52.068813 7fd5798fd700 4 mds.0.purge_queue operator(): data pool 6 not found in OSDMap
-145> 2018-03-22 15:52:52.069016 7fd5798fd700 5 asok(0x555883c661e0) register_command objecter_requests hook 0x555883c362e0
-144> 2018-03-22 15:52:52.069078 7fd5798fd700 10 monclient: _renew_subs
-143> 2018-03-22 15:52:52.069090 7fd5798fd700 10 monclient: _send_mon_message to mon.filer1 at x.x.1.17:6789/0
-142> 2018-03-22 15:52:52.069118 7fd5798fd700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6789/0 -- mon_subscribe({osdmap=0}) v2 -- 0x555883c8f680 con 0
-141> 2018-03-22 15:52:52.069199 7fd5798fd700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201)
-140> 2018-03-22 15:52:52.069430 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6789/0 conn(0x555883eb5000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=793748 cs=1 l=1). rx mon.0 seq 12 0x555883df3900 osd_map(1396..1396 src has 752..1396) v3
-139> 2018-03-22 15:52:52.069523 7fd5798fd700 4 mds.0.purge_queue operator(): data pool 6 not found in OSDMap
-138> 2018-03-22 15:52:52.069538 7fd5798fd700 4 mds.0.0 handle_osd_map epoch 0, 0 new blacklist entries
-137> 2018-03-22 15:52:52.069774 7fd5798fd700 1 mds.0.5325 handle_mds_map i am now mds.0.5325
-136> 2018-03-22 15:52:52.069784 7fd5798fd700 1 mds.0.5325 handle_mds_map state change up:boot --> up:replay
-135> 2018-03-22 15:52:52.069810 7fd5798fd700 1 mds.0.5325 replay_start
-134> 2018-03-22 15:52:52.069821 7fd5798fd700 1 mds.0.5325 recovery set is
-133> 2018-03-22 15:52:52.069845 7fd5798fd700 1 mds.0.5325 waiting for osdmap 1396 (which blacklists prior instance)
-132> 2018-03-22 15:52:52.069855 7fd5798fd700 4 mds.0.purge_queue operator(): data pool 6 not found in OSDMap
-131> 2018-03-22 15:52:52.069888 7fd5798fd700 1 -- x.x.1.17:6803/122963511 <== mon.0 x.x.1.17:6789/0 12 ==== osd_map(1396..1396 src has 752..1396) v3 ==== 3162+0+0 (2330026281 0 0) 0x555883df3900 con 0x555883eb5000
-130> 2018-03-22 15:52:52.070103 7fd5798fd700 4 mds.0.5325 handle_osd_map epoch 1396, 0 new blacklist entries
-129> 2018-03-22 15:52:52.070119 7fd5798fd700 10 monclient: _renew_subs
-128> 2018-03-22 15:52:52.070125 7fd5798fd700 10 monclient: _send_mon_message to mon.filer1 at x.x.1.17:6789/0
-127> 2018-03-22 15:52:52.070147 7fd5798fd700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6789/0 -- mon_subscribe({osdmap=1397}) v2 -- 0x555883c8efc0 con 0
-126> 2018-03-22 15:52:52.070243 7fd5738f1700 2 mds.0.5325 boot_start 0: opening inotable
-125> 2018-03-22 15:52:52.070422 7fd5738f1700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6801/61085 -- osd_op(unknown.0.5325:1 7.13 7:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f64000 con 0
-124> 2018-03-22 15:52:52.070473 7fd5738f1700 2 mds.0.5325 boot_start 0: opening sessionmap
-123> 2018-03-22 15:52:52.070555 7fd5738f1700 1 -- x.x.1.17:6803/122963511 --> x.x.1.18:6801/2025 -- osd_op(unknown.0.5325:2 7.b 7:d0630e4c:::mds0_sessionmap:head [omap-get-header,omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f64340 con 0
-122> 2018-03-22 15:52:52.070585 7fd5738f1700 2 mds.0.5325 boot_start 0: opening mds log
-121> 2018-03-22 15:52:52.070593 7fd5738f1700 5 mds.0.log open discovering log bounds
-120> 2018-03-22 15:52:52.070718 7fd5738f1700 2 mds.0.5325 boot_start 0: opening purge queue (async)
-119> 2018-03-22 15:52:52.070733 7fd5738f1700 4 mds.0.purge_queue open: opening
-118> 2018-03-22 15:52:52.070736 7fd5738f1700 1 mds.0.journaler.pq(ro) recover start
-117> 2018-03-22 15:52:52.070739 7fd5738f1700 1 mds.0.journaler.pq(ro) read_head
-116> 2018-03-22 15:52:52.070744 7fd5730f0700 4 mds.0.journalpointer Reading journal pointer '400.00000000'
-115> 2018-03-22 15:52:52.070759 7fd5738f1700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6801/61085 -- osd_op(unknown.0.5325:3 7.55 7:aa448500:::500.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f64680 con 0
-114> 2018-03-22 15:52:52.070799 7fd5730f0700 1 -- x.x.1.17:6803/122963511 --> x.x.1.18:6801/2025 -- osd_op(unknown.0.5325:4 7.f 7:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f649c0 con 0
-113> 2018-03-22 15:52:52.070832 7fd5738f1700 2 mds.0.5325 boot_start 0: opening snap table
-112> 2018-03-22 15:52:52.070876 7fd5738f1700 1 -- x.x.1.17:6803/122963511 --> x.x.1.19:6801/56955 -- osd_op(unknown.0.5325:5 7.2d 7:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f64d00 con 0
-111> 2018-03-22 15:52:52.071154 7fd57c8a0700 2 -- x.x.1.17:6803/122963511 >> x.x.1.17:6801/61085 conn(0x555883f5a000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
-110> 2018-03-22 15:52:52.071287 7fd57c09f700 2 -- x.x.1.17:6803/122963511 >> x.x.1.18:6801/2025 conn(0x555883f5b800 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
-109> 2018-03-22 15:52:52.071525 7fd57b89e700 2 -- x.x.1.17:6803/122963511 >> x.x.1.19:6801/56955 conn(0x555883f5d000 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=1)._process_connection got newly_acked_seq 0 vs out_seq 0
-108> 2018-03-22 15:52:52.071941 7fd57b89e700 5 -- x.x.1.17:6803/122963511 >> x.x.1.19:6801/56955 conn(0x555883f5d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=111 cs=1 l=1). rx osd.2 seq 1 0x555883f64d00 osd_op_reply(5 mds_snaptable [read 0~46] v0'0 uv1 ondisk = 0) v8
-107> 2018-03-22 15:52:52.071968 7fd57b89e700 1 -- x.x.1.17:6803/122963511 <== osd.2 x.x.1.19:6801/56955 1 ==== osd_op_reply(5 mds_snaptable [read 0~46] v0'0 uv1 ondisk = 0) v8 ==== 157+0+46 (367977290 0 2603725198) 0x555883f64d00 con 0x555883f5d000
-106> 2018-03-22 15:52:52.071951 7fd57c8a0700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6801/61085 conn(0x555883f5a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=117 cs=1 l=1). rx osd.0 seq 1 0x555883f64340 osd_op_reply(3 500.00000000 [read 0~90] v0'0 uv142815 ondisk = 0) v8
-105> 2018-03-22 15:52:52.072004 7fd57c8a0700 1 -- x.x.1.17:6803/122963511 <== osd.0 x.x.1.17:6801/61085 1 ==== osd_op_reply(3 500.00000000 [read 0~90] v0'0 uv142815 ondisk = 0) v8 ==== 156+0+90 (4167717821 0 1992813580) 0x555883f64340 con 0x555883f5a000
-104> 2018-03-22 15:52:52.072062 7fd5748f3700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 12582912, expire 13845209, write 13845209, stream_format 1). probing for end of log (from 13845209)...
-103> 2018-03-22 15:52:52.072074 7fd5748f3700 1 mds.0.journaler.pq(ro) probing for end of the log
-102> 2018-03-22 15:52:52.072123 7fd5748f3700 1 -- x.x.1.17:6803/122963511 --> x.x.1.19:6801/56955 -- osd_op(unknown.0.5325:6 7.34 7:2e992e0f:::500.00000003:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+full_force e1396) v8 -- 0x555883f64000 con 0
-101> 2018-03-22 15:52:52.072213 7fd5748f3700 1 -- x.x.1.17:6803/122963511 --> x.x.1.18:6801/2025 -- osd_op(unknown.0.5325:7 7.26 7:66eb7180:::500.00000004:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+full_force e1396) v8 -- 0x555883f64680 con 0
-100> 2018-03-22 15:52:52.072690 7fd57b89e700 5 -- x.x.1.17:6803/122963511 >> x.x.1.19:6801/56955 conn(0x555883f5d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=111 cs=1 l=1). rx osd.2 seq 2 0x555883f64d00 osd_op_reply(6 500.00000003 [stat] v0'0 uv282976 ondisk = 0) v8
-99> 2018-03-22 15:52:52.072712 7fd57b89e700 1 -- x.x.1.17:6803/122963511 <== osd.2 x.x.1.19:6801/56955 2 ==== osd_op_reply(6 500.00000003 [stat] v0'0 uv282976 ondisk = 0) v8 ==== 156+0+16 (3036645653 0 1497514102) 0x555883f64d00 con 0x555883f5d000
-98> 2018-03-22 15:52:52.285269 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.18:6801/2025 conn(0x555883f5b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=110 cs=1 l=1). rx osd.1 seq 1 0x555883f64680 osd_op_reply(2 mds0_sessionmap [omap-get-header,omap-get-vals] v0'0 uv158730 ondisk = 0) v8
-97> 2018-03-22 15:52:52.285301 7fd57c09f700 1 -- x.x.1.17:6803/122963511 <== osd.1 x.x.1.18:6801/2025 1 ==== osd_op_reply(2 mds0_sessionmap [omap-get-header,omap-get-vals] v0'0 uv158730 ondisk = 0) v8 ==== 201+0+16343 (2396657306 0 2795133511) 0x555883f64680 con 0x555883f5b800
-96> 2018-03-22 15:52:52.290264 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.18:6801/2025 conn(0x555883f5b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=110 cs=1 l=1). rx osd.1 seq 2 0x555883f64680 osd_op_reply(7 500.00000004 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8
-95> 2018-03-22 15:52:52.290300 7fd57c09f700 1 -- x.x.1.17:6803/122963511 <== osd.1 x.x.1.18:6801/2025 2 ==== osd_op_reply(7 500.00000004 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (2305979436 0 0) 0x555883f64680 con 0x555883f5b800
-94> 2018-03-22 15:52:52.290376 7fd5748f3700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 13845209 (header had 13845209). recovered.
-93> 2018-03-22 15:52:52.290397 7fd5748f3700 4 mds.0.purge_queue operator(): open complete
-92> 2018-03-22 15:52:52.290402 7fd5748f3700 1 mds.0.journaler.pq(ro) set_writeable
-91> 2018-03-22 15:52:52.290899 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.18:6801/2025 conn(0x555883f5b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=110 cs=1 l=1). rx osd.1 seq 3 0x555883f64680 osd_op_reply(4 400.00000000 [read 0~22] v0'0 uv1 ondisk = 0) v8
-90> 2018-03-22 15:52:52.290928 7fd57c09f700 1 -- x.x.1.17:6803/122963511 <== osd.1 x.x.1.18:6801/2025 3 ==== osd_op_reply(4 400.00000000 [read 0~22] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (657553539 0 3825830296) 0x555883f64680 con 0x555883f5b800
-89> 2018-03-22 15:52:52.291057 7fd5730f0700 1 mds.0.journaler.mdlog(ro) recover start
-88> 2018-03-22 15:52:52.291068 7fd5730f0700 1 mds.0.journaler.mdlog(ro) read_head
-87> 2018-03-22 15:52:52.291098 7fd5730f0700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6801/61085 -- osd_op(unknown.0.5325:8 7.14 7:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f65a00 con 0
-86> 2018-03-22 15:52:52.291138 7fd5730f0700 4 mds.0.log Waiting for journal 0x200 to recover...
-85> 2018-03-22 15:52:52.291677 7fd57c8a0700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6801/61085 conn(0x555883f5a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=117 cs=1 l=1). rx osd.0 seq 2 0x555883f65a00 osd_op_reply(8 200.00000000 [read 0~90] v0'0 uv261722 ondisk = 0) v8
-84> 2018-03-22 15:52:52.291707 7fd57c8a0700 1 -- x.x.1.17:6803/122963511 <== osd.0 x.x.1.17:6801/61085 2 ==== osd_op_reply(8 200.00000000 [read 0~90] v0'0 uv261722 ondisk = 0) v8 ==== 156+0+90 (4187966182 0 1006404943) 0x555883f65a00 con 0x555883f5a000
-83> 2018-03-22 15:52:52.291754 7fd5738f1700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 219550842880, expire 219552551048, write 219615507873, stream_format 1). probing for end of log (from 219615507873)...
-82> 2018-03-22 15:52:52.291771 7fd5738f1700 1 mds.0.journaler.mdlog(ro) probing for end of the log
-81> 2018-03-22 15:52:52.291847 7fd5738f1700 1 -- x.x.1.17:6803/122963511 --> x.x.1.18:6801/2025 -- osd_op(unknown.0.5325:9 7.18 7:19465a45:::200.0000cc88:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+full_force e1396) v8 -- 0x555883f65380 con 0
-80> 2018-03-22 15:52:52.291906 7fd5738f1700 1 -- x.x.1.17:6803/122963511 --> x.x.1.18:6801/2025 -- osd_op(unknown.0.5325:10 7.2 7:41bb3fc2:::200.0000cc89:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+full_force e1396) v8 -- 0x555883f66080 con 0
-79> 2018-03-22 15:52:52.292473 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.18:6801/2025 conn(0x555883f5b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=110 cs=1 l=1). rx osd.1 seq 4 0x555883f64680 osd_op_reply(9 200.0000cc88 [stat] v0'0 uv120158 ondisk = 0) v8
-78> 2018-03-22 15:52:52.292514 7fd57c09f700 1 -- x.x.1.17:6803/122963511 <== osd.1 x.x.1.18:6801/2025 4 ==== osd_op_reply(9 200.0000cc88 [stat] v0'0 uv120158 ondisk = 0) v8 ==== 156+0+16 (2414739881 0 2839007350) 0x555883f64680 con 0x555883f5b800
-77> 2018-03-22 15:52:52.292570 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.18:6801/2025 conn(0x555883f5b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=110 cs=1 l=1). rx osd.1 seq 5 0x555883f64680 osd_op_reply(10 200.0000cc89 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8
-76> 2018-03-22 15:52:52.292592 7fd57c09f700 1 -- x.x.1.17:6803/122963511 <== osd.1 x.x.1.18:6801/2025 5 ==== osd_op_reply(10 200.0000cc89 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (1518717806 0 0) 0x555883f64680 con 0x555883f5b800
-75> 2018-03-22 15:52:52.292637 7fd5738f1700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 219615508275 (header had 219615507873). recovered.
-74> 2018-03-22 15:52:52.292712 7fd5730f0700 4 mds.0.log Journal 0x200 recovered.
-73> 2018-03-22 15:52:52.292727 7fd5730f0700 4 mds.0.log Recovered journal 0x200 in format 1
-72> 2018-03-22 15:52:52.315373 7fd5778f9700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6800/63308 -- mgrreport(unknown.filer1 +24-0 packed 214) v5 -- 0x555883c8be00 con 0
-71> 2018-03-22 15:52:52.478646 7fd57c8a0700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6801/61085 conn(0x555883f5a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=117 cs=1 l=1). rx osd.0 seq 3 0x555883f65a00 osd_op_reply(1 mds0_inotable [read 0~34] v0'0 uv143709 ondisk = 0) v8
-70> 2018-03-22 15:52:52.478690 7fd57c8a0700 1 -- x.x.1.17:6803/122963511 <== osd.0 x.x.1.17:6801/61085 3 ==== osd_op_reply(1 mds0_inotable [read 0~34] v0'0 uv143709 ondisk = 0) v8 ==== 157+0+34 (3459761511 0 469722454) 0x555883f65a00 con 0x555883f5a000
-69> 2018-03-22 15:52:52.478757 7fd5738f1700 2 mds.0.5325 boot_start 1: loading/discovering base inodes
-68> 2018-03-22 15:52:52.478789 7fd5738f1700 0 mds.0.cache creating system inode with ino:0x100
-67> 2018-03-22 15:52:52.478860 7fd5738f1700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6801/61085 -- osd_op(unknown.0.5325:11 7.33 7:cd5a64a3:::100.00000000:head [getxattr inode] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f663c0 con 0
-66> 2018-03-22 15:52:52.478916 7fd5738f1700 1 -- x.x.1.17:6803/122963511 --> x.x.1.19:6801/56955 -- osd_op(unknown.0.5325:12 7.3f 7:fe07bba1:::100.00000000.inode:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f66700 con 0
-65> 2018-03-22 15:52:52.478966 7fd5738f1700 0 mds.0.cache creating system inode with ino:0x1
-64> 2018-03-22 15:52:52.479063 7fd5738f1700 1 -- x.x.1.17:6803/122963511 --> x.x.1.19:6801/56955 -- osd_op(unknown.0.5325:13 7.3f 7:ff5b34d6:::1.00000000:head [getxattr inode] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f66a40 con 0
-63> 2018-03-22 15:52:52.479123 7fd5738f1700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6801/61085 -- osd_op(unknown.0.5325:14 7.14 7:287034c4:::1.00000000.inode:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f649c0 con 0
-62> 2018-03-22 15:52:52.479479 7fd57c8a0700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6801/61085 conn(0x555883f5a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=117 cs=1 l=1). rx osd.0 seq 4 0x555883f649c0 osd_op_reply(11 100.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8
-61> 2018-03-22 15:52:52.479506 7fd57b89e700 5 -- x.x.1.17:6803/122963511 >> x.x.1.19:6801/56955 conn(0x555883f5d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=111 cs=1 l=1). rx osd.2 seq 3 0x555883f64000 osd_op_reply(12 100.00000000.inode [read 0~536] v0'0 uv304042 ondisk = 0) v8
-60> 2018-03-22 15:52:52.479533 7fd57b89e700 1 -- x.x.1.17:6803/122963511 <== osd.2 x.x.1.19:6801/56955 3 ==== osd_op_reply(12 100.00000000.inode [read 0~536] v0'0 uv304042 ondisk = 0) v8 ==== 162+0+536 (4230129339 0 4167152603) 0x555883f64000 con 0x555883f5d000
-59> 2018-03-22 15:52:52.479524 7fd57c8a0700 1 -- x.x.1.17:6803/122963511 <== osd.0 x.x.1.17:6801/61085 4 ==== osd_op_reply(11 100.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 156+0+0 (802953997 0 0) 0x555883f649c0 con 0x555883f5a000
-58> 2018-03-22 15:52:52.479576 7fd57b89e700 5 -- x.x.1.17:6803/122963511 >> x.x.1.19:6801/56955 conn(0x555883f5d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=111 cs=1 l=1). rx osd.2 seq 4 0x555883f64000 osd_op_reply(13 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8
-57> 2018-03-22 15:52:52.479592 7fd57b89e700 1 -- x.x.1.17:6803/122963511 <== osd.2 x.x.1.19:6801/56955 4 ==== osd_op_reply(13 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 154+0+0 (2478131573 0 0) 0x555883f64000 con 0x555883f5d000
-56> 2018-03-22 15:52:52.479712 7fd57c8a0700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6801/61085 conn(0x555883f5a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=117 cs=1 l=1). rx osd.0 seq 5 0x555883f649c0 osd_op_reply(14 1.00000000.inode [read 0~536] v0'0 uv261708 ondisk = 0) v8
-55> 2018-03-22 15:52:52.479731 7fd57c8a0700 1 -- x.x.1.17:6803/122963511 <== osd.0 x.x.1.17:6801/61085 5 ==== osd_op_reply(14 1.00000000.inode [read 0~536] v0'0 uv261708 ondisk = 0) v8 ==== 160+0+536 (3842073863 0 828467972) 0x555883f649c0 con 0x555883f5a000
-54> 2018-03-22 15:52:52.479769 7fd5738f1700 2 mds.0.5325 boot_start 2: replaying mds log
-53> 2018-03-22 15:52:52.479783 7fd5738f1700 2 mds.0.5325 boot_start 2: waiting for purge queue recovered
-52> 2018-03-22 15:52:52.479906 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.18:6801/2025 -- osd_op(unknown.0.5325:15 7.3a 7:5d17787c:::200.0000cc79:head [read 1708168~2486136 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f670c0 con 0
-51> 2018-03-22 15:52:52.480004 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.19:6801/56955 -- osd_op(unknown.0.5325:16 7.2f 7:f43cbe1a:::200.0000cc7a:head [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f67400 con 0
-50> 2018-03-22 15:52:52.480059 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.19:6801/56955 -- osd_op(unknown.0.5325:17 7.5 7:a1dc1a9d:::200.0000cc7b:head [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f67740 con 0
-49> 2018-03-22 15:52:52.480109 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.18:6801/2025 -- osd_op(unknown.0.5325:18 7.0 7:00cc040d:::200.0000cc7c:head [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f67a80 con 0
-48> 2018-03-22 15:52:52.480192 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.19:6801/56955 -- osd_op(unknown.0.5325:19 7.2d 7:b5788a65:::200.0000cc7d:head [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f66a40 con 0
-47> 2018-03-22 15:52:52.480242 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6801/61085 -- osd_op(unknown.0.5325:20 7.43 7:c359db68:::200.0000cc7e:head [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f66700 con 0
-46> 2018-03-22 15:52:52.480306 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.18:6801/2025 -- osd_op(unknown.0.5325:21 7.36 7:6d91f983:::200.0000cc7f:head [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f64d00 con 0
-45> 2018-03-22 15:52:52.480351 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6801/61085 -- osd_op(unknown.0.5325:22 7.8 7:1168fae1:::200.0000cc80:head [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f64000 con 0
-44> 2018-03-22 15:52:52.480395 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6801/61085 -- osd_op(unknown.0.5325:23 7.30 7:0f8e423f:::200.0000cc81:head [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f66080 con 0
-43> 2018-03-22 15:52:52.480437 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6801/61085 -- osd_op(unknown.0.5325:24 7.2a 7:553af0e4:::200.0000cc82:head [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f65380 con 0
-42> 2018-03-22 15:52:52.480493 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.18:6801/2025 -- osd_op(unknown.0.5325:25 7.f 7:f136b8d6:::200.0000cc83:head [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f64680 con 0
-41> 2018-03-22 15:52:52.509055 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.18:6801/2025 conn(0x555883f5b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=110 cs=1 l=1). rx osd.1 seq 6 0x555883f64680 osd_op_reply(18 200.0000cc7c [read 0~4194304 [fadvise_dontneed]] v0'0 uv111922 ondisk = 0) v8
-40> 2018-03-22 15:52:52.509100 7fd57c09f700 1 -- x.x.1.17:6803/122963511 <== osd.1 x.x.1.18:6801/2025 6 ==== osd_op_reply(18 200.0000cc7c [read 0~4194304 [fadvise_dontneed]] v0'0 uv111922 ondisk = 0) v8 ==== 156+0+4194304 (2889947030 0 2232087157) 0x555883f64680 con 0x555883f5b800
-39> 2018-03-22 15:52:52.511367 7fd57c8a0700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6801/61085 conn(0x555883f5a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=117 cs=1 l=1). rx osd.0 seq 6 0x555883f649c0 osd_op_reply(22 200.0000cc80 [read 0~4194304 [fadvise_dontneed]] v0'0 uv122611 ondisk = 0) v8
-38> 2018-03-22 15:52:52.511416 7fd57c8a0700 1 -- x.x.1.17:6803/122963511 <== osd.0 x.x.1.17:6801/61085 6 ==== osd_op_reply(22 200.0000cc80 [read 0~4194304 [fadvise_dontneed]] v0'0 uv122611 ondisk = 0) v8 ==== 156+0+4194304 (360001783 0 3146556651) 0x555883f649c0 con 0x555883f5a000
-37> 2018-03-22 15:52:52.514358 7fd57c8a0700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6801/61085 conn(0x555883f5a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=117 cs=1 l=1). rx osd.0 seq 7 0x555883f649c0 osd_op_reply(20 200.0000cc7e [read 0~4194304 [fadvise_dontneed]] v0'0 uv133445 ondisk = 0) v8
-36> 2018-03-22 15:52:52.514389 7fd57c8a0700 1 -- x.x.1.17:6803/122963511 <== osd.0 x.x.1.17:6801/61085 7 ==== osd_op_reply(20 200.0000cc7e [read 0~4194304 [fadvise_dontneed]] v0'0 uv133445 ondisk = 0) v8 ==== 156+0+4194304 (2090475696 0 2923532206) 0x555883f649c0 con 0x555883f5a000
-35> 2018-03-22 15:52:52.515913 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.18:6801/2025 conn(0x555883f5b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=110 cs=1 l=1). rx osd.1 seq 7 0x555883f64680 osd_op_reply(15 200.0000cc79 [read 1708168~2486136 [fadvise_dontneed]] v0'0 uv267351 ondisk = 0) v8
-34> 2018-03-22 15:52:52.516279 7fd57c09f700 1 -- x.x.1.17:6803/122963511 <== osd.1 x.x.1.18:6801/2025 7 ==== osd_op_reply(15 200.0000cc79 [read 1708168~2486136 [fadvise_dontneed]] v0'0 uv267351 ondisk = 0) v8 ==== 156+0+2486136 (2310531997 0 1366848730) 0x555883f64680 con 0x555883f5b800
-33> 2018-03-22 15:52:52.519771 7fd57c8a0700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6801/61085 conn(0x555883f5a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=117 cs=1 l=1). rx osd.0 seq 8 0x555883f649c0 osd_op_reply(24 200.0000cc82 [read 0~4194304 [fadvise_dontneed]] v0'0 uv270305 ondisk = 0) v8
-32> 2018-03-22 15:52:52.519817 7fd57c8a0700 1 -- x.x.1.17:6803/122963511 <== osd.0 x.x.1.17:6801/61085 8 ==== osd_op_reply(24 200.0000cc82 [read 0~4194304 [fadvise_dontneed]] v0'0 uv270305 ondisk = 0) v8 ==== 156+0+4194304 (3850178096 0 828145897) 0x555883f649c0 con 0x555883f5a000
-31> 2018-03-22 15:52:52.523971 7fd57b89e700 5 -- x.x.1.17:6803/122963511 >> x.x.1.19:6801/56955 conn(0x555883f5d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=111 cs=1 l=1). rx osd.2 seq 5 0x555883f66a40 osd_op_reply(17 200.0000cc7b [read 0~4194304 [fadvise_dontneed]] v0'0 uv128946 ondisk = 0) v8
-30> 2018-03-22 15:52:52.524006 7fd57b89e700 1 -- x.x.1.17:6803/122963511 <== osd.2 x.x.1.19:6801/56955 5 ==== osd_op_reply(17 200.0000cc7b [read 0~4194304 [fadvise_dontneed]] v0'0 uv128946 ondisk = 0) v8 ==== 156+0+4194304 (2098079074 0 3003574979) 0x555883f66a40 con 0x555883f5d000
-29> 2018-03-22 15:52:52.526237 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.18:6801/2025 conn(0x555883f5b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=110 cs=1 l=1). rx osd.1 seq 8 0x555883f64680 osd_op_reply(21 200.0000cc7f [read 0~4194304 [fadvise_dontneed]] v0'0 uv284169 ondisk = 0) v8
-28> 2018-03-22 15:52:52.526262 7fd57c09f700 1 -- x.x.1.17:6803/122963511 <== osd.1 x.x.1.18:6801/2025 8 ==== osd_op_reply(21 200.0000cc7f [read 0~4194304 [fadvise_dontneed]] v0'0 uv284169 ondisk = 0) v8 ==== 156+0+4194304 (3827053419 0 3788447424) 0x555883f64680 con 0x555883f5b800
-27> 2018-03-22 15:52:52.533474 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.18:6801/2025 conn(0x555883f5b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=110 cs=1 l=1). rx osd.1 seq 9 0x555883f64680 osd_op_reply(25 200.0000cc83 [read 0~4194304 [fadvise_dontneed]] v0'0 uv120894 ondisk = 0) v8
-26> 2018-03-22 15:52:52.533509 7fd57c09f700 1 -- x.x.1.17:6803/122963511 <== osd.1 x.x.1.18:6801/2025 9 ==== osd_op_reply(25 200.0000cc83 [read 0~4194304 [fadvise_dontneed]] v0'0 uv120894 ondisk = 0) v8 ==== 156+0+4194304 (1476261173 0 3169243193) 0x555883f64680 con 0x555883f5b800
-25> 2018-03-22 15:52:52.533943 7fd57c8a0700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6801/61085 conn(0x555883f5a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=117 cs=1 l=1). rx osd.0 seq 9 0x555883f649c0 osd_op_reply(23 200.0000cc81 [read 0~4194304 [fadvise_dontneed]] v0'0 uv232635 ondisk = 0) v8
-24> 2018-03-22 15:52:52.533982 7fd57c8a0700 1 -- x.x.1.17:6803/122963511 <== osd.0 x.x.1.17:6801/61085 9 ==== osd_op_reply(23 200.0000cc81 [read 0~4194304 [fadvise_dontneed]] v0'0 uv232635 ondisk = 0) v8 ==== 156+0+4194304 (3321494973 0 2672554052) 0x555883f649c0 con 0x555883f5a000
-23> 2018-03-22 15:52:52.534996 7fd57b89e700 5 -- x.x.1.17:6803/122963511 >> x.x.1.19:6801/56955 conn(0x555883f5d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=111 cs=1 l=1). rx osd.2 seq 6 0x555883f66a40 osd_op_reply(16 200.0000cc7a [read 0~4194304 [fadvise_dontneed]] v0'0 uv254456 ondisk = 0) v8
-22> 2018-03-22 15:52:52.535027 7fd57b89e700 1 -- x.x.1.17:6803/122963511 <== osd.2 x.x.1.19:6801/56955 6 ==== osd_op_reply(16 200.0000cc7a [read 0~4194304 [fadvise_dontneed]] v0'0 uv254456 ondisk = 0) v8 ==== 156+0+4194304 (2101888833 0 3301144556) 0x555883f66a40 con 0x555883f5d000
-21> 2018-03-22 15:52:52.535213 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.19:6801/56955 -- osd_op(unknown.0.5325:26 7.5 7:a0c3938f:::200.0000cc84:head [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555883f670c0 con 0
-20> 2018-03-22 15:52:52.541605 7fd57b89e700 5 -- x.x.1.17:6803/122963511 >> x.x.1.19:6801/56955 conn(0x555883f5d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=111 cs=1 l=1). rx osd.2 seq 7 0x555883f670c0 osd_op_reply(19 200.0000cc7d [read 0~4194304 [fadvise_dontneed]] v0'0 uv255276 ondisk = 0) v8
-19> 2018-03-22 15:52:52.541629 7fd57b89e700 1 -- x.x.1.17:6803/122963511 <== osd.2 x.x.1.19:6801/56955 7 ==== osd_op_reply(19 200.0000cc7d [read 0~4194304 [fadvise_dontneed]] v0'0 uv255276 ondisk = 0) v8 ==== 156+0+4194304 (561833161 0 2848452092) 0x555883f670c0 con 0x555883f5d000
-18> 2018-03-22 15:52:52.574486 7fd57b89e700 5 -- x.x.1.17:6803/122963511 >> x.x.1.19:6801/56955 conn(0x555883f5d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=111 cs=1 l=1). rx osd.2 seq 8 0x555883f670c0 osd_op_reply(26 200.0000cc84 [read 0~4194304 [fadvise_dontneed]] v0'0 uv128948 ondisk = 0) v8
-17> 2018-03-22 15:52:52.574508 7fd57b89e700 1 -- x.x.1.17:6803/122963511 <== osd.2 x.x.1.19:6801/56955 8 ==== osd_op_reply(26 200.0000cc84 [read 0~4194304 [fadvise_dontneed]] v0'0 uv128948 ondisk = 0) v8 ==== 156+0+4194304 (2539366010 0 517612525) 0x555883f670c0 con 0x555883f5d000
-16> 2018-03-22 15:52:52.644645 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.17:6801/61085 -- osd_op(unknown.0.5325:27 7.44 7:221f44ba:::200.0000cc85:head [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x55588859f740 con 0
-15> 2018-03-22 15:52:52.663434 7fd57c8a0700 5 -- x.x.1.17:6803/122963511 >> x.x.1.17:6801/61085 conn(0x555883f5a000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=117 cs=1 l=1). rx osd.0 seq 10 0x55588859f740 osd_op_reply(27 200.0000cc85 [read 0~4194304 [fadvise_dontneed]] v0'0 uv122793 ondisk = 0) v8
-14> 2018-03-22 15:52:52.663477 7fd57c8a0700 1 -- x.x.1.17:6803/122963511 <== osd.0 x.x.1.17:6801/61085 10 ==== osd_op_reply(27 200.0000cc85 [read 0~4194304 [fadvise_dontneed]] v0'0 uv122793 ondisk = 0) v8 ==== 156+0+4194304 (4175891152 0 992964831) 0x55588859f740 con 0x555883f5a000
-13> 2018-03-22 15:52:52.691398 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.19:6801/56955 -- osd_op(unknown.0.5325:28 7.19 7:999ffb8c:::200.0000cc86:head [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555887d6da80 con 0
-12> 2018-03-22 15:52:52.709210 7fd57b89e700 5 -- x.x.1.17:6803/122963511 >> x.x.1.19:6801/56955 conn(0x555883f5d000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=111 cs=1 l=1). rx osd.2 seq 9 0x555887d6da80 osd_op_reply(28 200.0000cc86 [read 0~4194304 [fadvise_dontneed]] v0'0 uv133938 ondisk = 0) v8
-11> 2018-03-22 15:52:52.709280 7fd57b89e700 1 -- x.x.1.17:6803/122963511 <== osd.2 x.x.1.19:6801/56955 9 ==== osd_op_reply(28 200.0000cc86 [read 0~4194304 [fadvise_dontneed]] v0'0 uv133938 ondisk = 0) v8 ==== 156+0+4194304 (3792192724 0 3482327516) 0x555887d6da80 con 0x555883f5d000
-10> 2018-03-22 15:52:52.732088 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.18:6801/2025 -- osd_op(unknown.0.5325:29 7.58 7:1b7f2330:::200.0000cc87:head [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555886c63a00 con 0
-9> 2018-03-22 15:52:52.776011 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.18:6801/2025 conn(0x555883f5b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=110 cs=1 l=1). rx osd.1 seq 10 0x555886c63a00 osd_op_reply(29 200.0000cc87 [read 0~4194304 [fadvise_dontneed]] v0'0 uv123301 ondisk = 0) v8
-8> 2018-03-22 15:52:52.776055 7fd57c09f700 1 -- x.x.1.17:6803/122963511 <== osd.1 x.x.1.18:6801/2025 10 ==== osd_op_reply(29 200.0000cc87 [read 0~4194304 [fadvise_dontneed]] v0'0 uv123301 ondisk = 0) v8 ==== 156+0+4194304 (3322835623 0 1085409543) 0x555886c63a00 con 0x555883f5b800
-7> 2018-03-22 15:52:52.807570 7fd5720ee700 1 -- x.x.1.17:6803/122963511 --> x.x.1.18:6801/2025 -- osd_op(unknown.0.5325:30 7.18 7:19465a45:::200.0000cc88:head [read 0~1750835 [fadvise_dontneed]] snapc 0=[] ondisk+read+known_if_redirected+full_force e1396) v8 -- 0x555886ff2680 con 0
-6> 2018-03-22 15:52:52.865657 7fd57c09f700 5 -- x.x.1.17:6803/122963511 >> x.x.1.18:6801/2025 conn(0x555883f5b800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=110 cs=1 l=1). rx osd.1 seq 11 0x555886ff2680 osd_op_reply(30 200.0000cc88 [read 0~1750835 [fadvise_dontneed]] v0'0 uv120158 ondisk = 0) v8
-5> 2018-03-22 15:52:52.865702 7fd57c09f700 1 -- x.x.1.17:6803/122963511 <== osd.1 x.x.1.18:6801/2025 11 ==== osd_op_reply(30 200.0000cc88 [read 0~1750835 [fadvise_dontneed]] v0'0 uv120158 ondisk = 0) v8 ==== 156+0+1750835 (3927299451 0 1933165720) 0x555886ff2680 con 0x555883f5b800
-4> 2018-03-22 15:52:53.231977 7fd5720ee700 -1 log_channel(cluster) log [ERR] : journal replay inotablev mismatch 1 -> 53668
-3> 2018-03-22 15:52:53.273998 7fd5720ee700 -1 log_channel(cluster) log [ERR] : Client session loaded with invalid preallocated inodes, evicting session r-laptop:nodes (705417)
-2> 2018-03-22 15:52:53.274027 7fd5720ee700 -1 log_channel(cluster) log [ERR] : Client session loaded with invalid preallocated inodes, evicting session node01:nodes (724115)
-1> 2018-03-22 15:52:53.274030 7fd5720ee700 -1 log_channel(cluster) log [ERR] : Client session loaded with invalid preallocated inodes, evicting session r-laptop:nodes (724169)
0> 2018-03-22 15:52:53.275454 7fd5720ee700 -1 /build/ceph-12.2.4/src/mds/MDLog.cc: In function 'void MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)' thread 7fd5720ee700 time 2018-03-22 15:52:53.274039
/build/ceph-12.2.4/src/mds/MDLog.cc: 275: FAILED assert(!mds->is_any_replay())
ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x555879e88942]
2: (MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0x567) [0x555879ddd4e7]
3: (Server::journal_close_session(Session*, int, Context*)+0x963) [0x555879b8d603]
4: (Server::kill_session(Session*, Context*)+0x1fd) [0x555879b8e56d]
5: (MDSRank::validate_sessions()+0x2dc) [0x555879b4c2dc]
6: (MDSRank::boot_start(MDSRank::BootStep, int)+0xc08) [0x555879b4d228]
7: (MDSInternalContextBase::complete(int)+0x18b) [0x555879dc56db]
8: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::sub_finish(MDSInternalContextBase*, int)+0x127) [0x555879b66627]
9: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::complete(int)+0x21) [0x555879b668d1]
10: (MDLog::_replay_thread()+0x43c) [0x555879ddac8c]
11: (MDLog::ReplayThread::entry()+0xd) [0x555879b56fcd]
12: (()+0x76ba) [0x7fd57f3f06ba]
13: (clone()+0x6d) [0x7fd57e45c41d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mds.filer1.log
--- end dump of recent events ---
2018-03-22 15:52:53.278049 7fd5720ee700 -1 *** Caught signal (Aborted) **
in thread 7fd5720ee700 thread_name:md_log_replay
ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
1: (()+0x5ab254) [0x555879e42254]
2: (()+0x11390) [0x7fd57f3fa390]
3: (gsignal()+0x38) [0x7fd57e38a428]
4: (abort()+0x16a) [0x7fd57e38c02a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x555879e88ace]
6: (MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0x567) [0x555879ddd4e7]
7: (Server::journal_close_session(Session*, int, Context*)+0x963) [0x555879b8d603]
8: (Server::kill_session(Session*, Context*)+0x1fd) [0x555879b8e56d]
9: (MDSRank::validate_sessions()+0x2dc) [0x555879b4c2dc]
10: (MDSRank::boot_start(MDSRank::BootStep, int)+0xc08) [0x555879b4d228]
11: (MDSInternalContextBase::complete(int)+0x18b) [0x555879dc56db]
12: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::sub_finish(MDSInternalContextBase*, int)+0x127) [0x555879b66627]
13: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::complete(int)+0x21) [0x555879b668d1]
14: (MDLog::_replay_thread()+0x43c) [0x555879ddac8c]
15: (MDLog::ReplayThread::entry()+0xd) [0x555879b56fcd]
16: (()+0x76ba) [0x7fd57f3f06ba]
17: (clone()+0x6d) [0x7fd57e45c41d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2018-03-22 15:52:53.278049 7fd5720ee700 -1 *** Caught signal (Aborted) **
in thread 7fd5720ee700 thread_name:md_log_replay
ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
1: (()+0x5ab254) [0x555879e42254]
2: (()+0x11390) [0x7fd57f3fa390]
3: (gsignal()+0x38) [0x7fd57e38a428]
4: (abort()+0x16a) [0x7fd57e38c02a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x555879e88ace]
6: (MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0x567) [0x555879ddd4e7]
7: (Server::journal_close_session(Session*, int, Context*)+0x963) [0x555879b8d603]
8: (Server::kill_session(Session*, Context*)+0x1fd) [0x555879b8e56d]
9: (MDSRank::validate_sessions()+0x2dc) [0x555879b4c2dc]
10: (MDSRank::boot_start(MDSRank::BootStep, int)+0xc08) [0x555879b4d228]
11: (MDSInternalContextBase::complete(int)+0x18b) [0x555879dc56db]
12: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::sub_finish(MDSInternalContextBase*, int)+0x127) [0x555879b66627]
13: (C_GatherBase<MDSInternalContextBase, MDSInternalContextGather>::C_GatherSub::complete(int)+0x21) [0x555879b668d1]
14: (MDLog::_replay_thread()+0x43c) [0x555879ddac8c]
15: (MDLog::ReplayThread::entry()+0xd) [0x555879b56fcd]
16: (()+0x76ba) [0x7fd57f3f06ba]
17: (clone()+0x6d) [0x7fd57e45c41d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mds.filer1.log
--- end dump of recent events ---
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
