Re: Cannot mount CephFS after irreversible OSD lost

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Dear John,

Thanks for such a prompt reply!

Seems like something happens on the mon side, since there are no mount-specific requests logged on the mds side (see below).
FYI, some hours ago I've disabled auth completely, but it didn't help.

The serialized metadata pool is 9.7G. I can try to compress it with 7z, then setup rssh account for you to scp/rsync it.

debug mds = 20
debug mon = 20

grep CLI.ENT.IPA.DDR /var/log/ceph/ceph-mon.000-s-ragnarok.log

2015-11-17 12:46:20.763049 7ffa90d11700 10 mon.000-s-ragnarok@0(leader) e1 ms_verify_authorizer xxx.xxx.xxx.xxx:0/137313644 client protocol 0
2015-11-17 12:46:20.763687 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader) e1 _ms_dispatch new session 0x5602b5178840 MonSession(unknown.0 xxx.xxx.xxx.xxx:0/137313644 is open)
2015-11-17 12:46:20.763699 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1  caps
2015-11-17 12:46:20.763720 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader).auth v5435 preprocess_query auth(proto 0 34 bytes epoch 0) from unknown.0 xxx.xxx.xxx.xxx:0/137313644
2015-11-17 12:46:20.763726 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader).auth v5435 prep_auth() blob_size=34
2015-11-17 12:46:20.763738 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader).auth v5435 AuthMonitor::assign_global_id m=auth(proto 0 34 bytes epoch 0) mon=0/1 last_allocated=1614103 max_global_id=1624096
2015-11-17 12:46:20.763741 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader).auth v5435 next_global_id should be 1614104
2015-11-17 12:46:20.763817 7ffa8b2e7700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b5350920 0x5602b535a480 auth_reply(proto 2 0 (0) Success) v1
2015-11-17 12:46:20.764469 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1 _ms_dispatch existing session 0x5602b5178840 for unknown.0 xxx.xxx.xxx.xxx:0/137313644
2015-11-17 12:46:20.764475 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1  caps
2015-11-17 12:46:20.764492 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader).auth v5435 preprocess_query auth(proto 2 32 bytes epoch 0) from unknown.0 xxx.xxx.xxx.xxx:0/137313644
2015-11-17 12:46:20.764497 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader).auth v5435 prep_auth() blob_size=32
2015-11-17 12:46:20.764705 7ffa8b2e7700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b5350920 0x5602b535b680 auth_reply(proto 2 0 (0) Success) v1
2015-11-17 12:46:20.765279 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1 _ms_dispatch existing session 0x5602b5178840 for unknown.0 xxx.xxx.xxx.xxx:0/137313644
2015-11-17 12:46:20.765287 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1  caps allow *
2015-11-17 12:46:20.765303 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader).auth v5435 preprocess_query auth(proto 2 165 bytes epoch 0) from unknown.0 xxx.xxx.xxx.xxx:0/137313644
2015-11-17 12:46:20.765310 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader).auth v5435 prep_auth() blob_size=165
2015-11-17 12:46:20.765532 7ffa8b2e7700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b5350920 0x5602b535a000 auth_reply(proto 2 0 (0) Success) v1
2015-11-17 12:46:20.766113 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1 _ms_dispatch existing session 0x5602b5178840 for unknown.0 xxx.xxx.xxx.xxx:0/137313644

and then

2015-11-17 12:48:20.767152 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader) e1 ms_handle_reset 0x5602b5913b80 xxx.xxx.xxx.xxx:0/137313644
2015-11-17 12:48:20.767167 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader) e1 reset/close on session unknown.0 xxx.xxx.xxx.xxx:0/137313644
2015-11-17 12:48:20.767173 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader) e1 remove_session 0x5602b5178840 unknown.0 xxx.xxx.xxx.xxx:0/137313644

session-specific stuff

2015-11-17 12:46:20.763817 7ffa8b2e7700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b5350920 0x5602b535a480 auth_reply(proto 2 0 (0) Success) v1
2015-11-17 12:46:20.764705 7ffa8b2e7700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b5350920 0x5602b535b680 auth_reply(proto 2 0 (0) Success) v1
2015-11-17 12:46:20.765532 7ffa8b2e7700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b5350920 0x5602b535a000 auth_reply(proto 2 0 (0) Success) v1
2015-11-17 12:46:21.995713 7ffa8b2e7700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b5350920 0x5602b5278900 mdsbeacon(1614101/000-s-ragnarok up:active seq 184 v9429) v4
2015-11-17 12:46:23.039318 7ffa8d109700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b5350920 0x5602b5388800 pg_stats_ack(1 pgs tid 389) v1
2015-11-17 12:47:24.056767 7ffa8d109700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b5350920 0x5602b5357400 pg_stats_ack(1 pgs tid 337) v1
2015-11-17 12:47:50.082888 7ffa8d109700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b5350920 0x5602b5cd6400 pg_stats_ack(2 pgs tid 263) v1

2015-11-17 12:46:20.763687 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader) e1 _ms_dispatch new session 0x5602b5178840 MonSession(unknown.0 xxx.xxx.xxx.xxx:0/137313644 is open)
2015-11-17 12:46:20.764469 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1 _ms_dispatch existing session 0x5602b5178840 for unknown.0 xxx.xxx.xxx.xxx:0/137313644
2015-11-17 12:46:20.765279 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1 _ms_dispatch existing session 0x5602b5178840 for unknown.0 xxx.xxx.xxx.xxx:0/137313644
2015-11-17 12:46:20.766113 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1 _ms_dispatch existing session 0x5602b5178840 for unknown.0 xxx.xxx.xxx.xxx:0/137313644

2015-11-17 12:46:20.764705 7ffa8b2e7700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b5350920 0x5602b535b680 auth_reply(proto 2 0 (0) Success) v1

tail -n 40 /var/log/ceph/ceph-mds.000-s-ragnarok.log

2015-11-17 12:41:59.573524 7fbbb27ff700 10 mds.0.log _trim_expired_segments waiting for 1841488226436/1841503004303 to expire
2015-11-17 12:41:59.573570 7fbbb27ff700 15 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
2015-11-17 12:41:59.573598 7fbbb27ff700 10 mds.0.server find_idle_sessions.  laggy until 0.000000
2015-11-17 12:41:59.573607 7fbbb27ff700 10 mds.0.locker scatter_tick
2015-11-17 12:41:59.573610 7fbbb27ff700 20 mds.0.locker caps_tick 0 revoking caps
2015-11-17 12:41:59.573612 7fbbb27ff700 15 mds.0.bal tick last_sample now 2015-11-17 12:41:59.573612
2015-11-17 12:41:59.573616 7fbbb27ff700 10 mds.0.cache find_stale_fragment_freeze
2015-11-17 12:41:59.573619 7fbbb27ff700 10 mds.0.snap check_osd_map - version unchanged
2015-11-17 12:42:01.978405 7fbbb1ffe700 10 mds.beacon.000-s-ragnarok _send up:active seq 119
2015-11-17 12:42:01.979208 7fbbb5ac5700 10 mds.beacon.000-s-ragnarok handle_mds_beacon up:active seq 119 rtt 0.000768
2015-11-17 12:42:04.573081 7fbbb27ff700  7 mds.0.cache trim max=100000  cur=17
2015-11-17 12:42:04.573109 7fbbb27ff700 10 mds.0.cache trim_client_leases
2015-11-17 12:42:04.573734 7fbbb27ff700  2 mds.0.cache check_memory_usage total 338472, rss 57992, heap 52152, malloc 1493 mmap 0, baseline 52152, buffers 0, 0 / 19 inodes have caps, 0 caps, 0 caps per inode
2015-11-17 12:42:04.573761 7fbbb27ff700 10 mds.0.log trim 1 / 30 segments, 8 / -1 events, 0 (0) expiring, 0 (0) expired
2015-11-17 12:42:04.573766 7fbbb27ff700 10 mds.0.log _trim_expired_segments waiting for 1841488226436/1841503004303 to expire
2015-11-17 12:42:04.573804 7fbbb27ff700 15 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
2015-11-17 12:42:04.573829 7fbbb27ff700 10 mds.0.server find_idle_sessions.  laggy until 0.000000
2015-11-17 12:42:04.573835 7fbbb27ff700 10 mds.0.locker scatter_tick
2015-11-17 12:42:04.573837 7fbbb27ff700 20 mds.0.locker caps_tick 0 revoking caps
2015-11-17 12:42:04.573840 7fbbb27ff700 15 mds.0.bal tick last_sample now 2015-11-17 12:42:04.573839
2015-11-17 12:42:04.573861 7fbbb27ff700 15 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
2015-11-17 12:42:04.573880 7fbbb27ff700  5 mds.0.bal mds.0 epoch 46 load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
2015-11-17 12:42:04.573895 7fbbb27ff700 10 mds.0.cache find_stale_fragment_freeze
2015-11-17 12:42:04.573898 7fbbb27ff700 10 mds.0.snap check_osd_map - version unchanged
2015-11-17 12:42:05.978664 7fbbb1ffe700 10 mds.beacon.000-s-ragnarok _send up:active seq 120
2015-11-17 12:42:05.979538 7fbbb5ac5700 10 mds.beacon.000-s-ragnarok handle_mds_beacon up:active seq 120 rtt 0.000836
2015-11-17 12:42:09.573339 7fbbb27ff700  7 mds.0.cache trim max=100000  cur=17
2015-11-17 12:42:09.573386 7fbbb27ff700 10 mds.0.cache trim_client_leases
2015-11-17 12:42:09.574005 7fbbb27ff700  2 mds.0.cache check_memory_usage total 338472, rss 58616, heap 52152, malloc 1499 mmap 0, baseline 52152, buffers 0, 0 / 19 inodes have caps, 0 caps, 0 caps per inode
2015-11-17 12:42:09.574031 7fbbb27ff700 10 mds.0.log trim 1 / 30 segments, 8 / -1 events, 0 (0) expiring, 0 (0) expired
2015-11-17 12:42:09.574036 7fbbb27ff700 10 mds.0.log _trim_expired_segments waiting for 1841488226436/1841503004303 to expire
2015-11-17 12:42:09.574079 7fbbb27ff700 15 mds.0.bal get_load mdsload<[0,0 0]/[0,0 0], req 0, hr 0, qlen 0, cpu 0>
2015-11-17 12:42:09.574106 7fbbb27ff700 10 mds.0.server find_idle_sessions.  laggy until 0.000000
2015-11-17 12:42:09.574115 7fbbb27ff700 10 mds.0.locker scatter_tick
2015-11-17 12:42:09.574118 7fbbb27ff700 20 mds.0.locker caps_tick 0 revoking caps
2015-11-17 12:42:09.574121 7fbbb27ff700 15 mds.0.bal tick last_sample now 2015-11-17 12:42:09.574120
2015-11-17 12:42:09.574125 7fbbb27ff700 10 mds.0.cache find_stale_fragment_freeze
2015-11-17 12:42:09.574128 7fbbb27ff700 10 mds.0.snap check_osd_map - version unchanged
2015-11-17 12:42:09.978921 7fbbb1ffe700 10 mds.beacon.000-s-ragnarok _send up:active seq 121
2015-11-17 12:42:09.979766 7fbbb5ac5700 10 mds.beacon.000-s-ragnarok handle_mds_beacon up:active seq 121 rtt 0.000808

grep -i mds /var/log/ceph/ceph-mon.000-s-ragnarok.log | tail -n 40

2015-11-17 12:47:25.999631 7ffa8b2e7700 20  allow so far , doing grant allow profile mds
2015-11-17 12:47:25.999637 7ffa8b2e7700 12 mon.000-s-ragnarok@0(leader).mds e9429 preprocess_beacon mdsbeacon(1614101/000-s-ragnarok up:active seq 200 v9429) v4 from mds.0 xxx.xxx.xxx.xxx:6800/2411 compat={},rocom
pat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table}
2015-11-17 12:47:25.999652 7ffa8b2e7700 15 mon.000-s-ragnarok@0(leader).mds e9429 _note_beacon mdsbeacon(1614101/000-s-ragnarok up:active seq 200 v9429) v4 noting time
2015-11-17 12:47:25.999662 7ffa8b2e7700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b53511e0 0x5602b51a1600 mdsbeacon(1614101/000-s-ragnarok up:active seq 200 v9429) v4
2015-11-17 12:47:27.903708 7ffa8bae8700 10 mon.000-s-ragnarok@0(leader).mds e9429 e9429: 1/1/0 up {0=000-s-ragnarok=up:active}
2015-11-17 12:47:29.999708 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1 _ms_dispatch existing session 0x5602b5178680 for mds.? xxx.xxx.xxx.xxx:6800/2411
2015-11-17 12:47:29.999718 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1  caps allow profile mds
2015-11-17 12:47:29.999745 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader).mds e9429 preprocess_query mdsbeacon(1614101/000-s-ragnarok up:active seq 201 v9429) v4 from mds.0 xxx.xxx.xxx.xxx:6800/2411
2015-11-17 12:47:29.999758 7ffa8b2e7700 20 is_capable service=mds command= exec on cap allow profile mds
2015-11-17 12:47:29.999762 7ffa8b2e7700 20  allow so far , doing grant allow profile mds
2015-11-17 12:47:29.999769 7ffa8b2e7700 12 mon.000-s-ragnarok@0(leader).mds e9429 preprocess_beacon mdsbeacon(1614101/000-s-ragnarok up:active seq 201 v9429) v4 from mds.0 xxx.xxx.xxx.xxx:6800/2411 compat={},rocom
pat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table}
2015-11-17 12:47:29.999783 7ffa8b2e7700 15 mon.000-s-ragnarok@0(leader).mds e9429 _note_beacon mdsbeacon(1614101/000-s-ragnarok up:active seq 201 v9429) v4 noting time
2015-11-17 12:47:29.999793 7ffa8b2e7700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b534fb20 0x5602b51a0d00 mdsbeacon(1614101/000-s-ragnarok up:active seq 201 v9429) v4
2015-11-17 12:47:32.904227 7ffa8bae8700 10 mon.000-s-ragnarok@0(leader).mds e9429 e9429: 1/1/0 up {0=000-s-ragnarok=up:active}
2015-11-17 12:47:33.999873 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1 _ms_dispatch existing session 0x5602b5178680 for mds.? xxx.xxx.xxx.xxx:6800/2411
2015-11-17 12:47:33.999885 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1  caps allow profile mds
2015-11-17 12:47:33.999912 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader).mds e9429 preprocess_query mdsbeacon(1614101/000-s-ragnarok up:active seq 202 v9429) v4 from mds.0 xxx.xxx.xxx.xxx:6800/2411
2015-11-17 12:47:33.999923 7ffa8b2e7700 20 is_capable service=mds command= exec on cap allow profile mds
2015-11-17 12:47:33.999928 7ffa8b2e7700 20  allow so far , doing grant allow profile mds
2015-11-17 12:47:33.999936 7ffa8b2e7700 12 mon.000-s-ragnarok@0(leader).mds e9429 preprocess_beacon mdsbeacon(1614101/000-s-ragnarok up:active seq 202 v9429) v4 from mds.0 xxx.xxx.xxx.xxx:6800/2411 compat={},rocom
pat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table}
2015-11-17 12:47:33.999950 7ffa8b2e7700 15 mon.000-s-ragnarok@0(leader).mds e9429 _note_beacon mdsbeacon(1614101/000-s-ragnarok up:active seq 202 v9429) v4 noting time
2015-11-17 12:47:33.999960 7ffa8b2e7700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b5351e20 0x5602b51a0700 mdsbeacon(1614101/000-s-ragnarok up:active seq 202 v9429) v4
2015-11-17 12:47:37.904741 7ffa8bae8700 10 mon.000-s-ragnarok@0(leader).mds e9429 e9429: 1/1/0 up {0=000-s-ragnarok=up:active}
2015-11-17 12:47:38.000302 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1 _ms_dispatch existing session 0x5602b5178680 for mds.? xxx.xxx.xxx.xxx:6800/2411
2015-11-17 12:47:38.000312 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1  caps allow profile mds
2015-11-17 12:47:38.000338 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader).mds e9429 preprocess_query mdsbeacon(1614101/000-s-ragnarok up:active seq 203 v9429) v4 from mds.0 xxx.xxx.xxx.xxx:6800/2411
2015-11-17 12:47:38.000381 7ffa8b2e7700 20 is_capable service=mds command= exec on cap allow profile mds
2015-11-17 12:47:38.000386 7ffa8b2e7700 20  allow so far , doing grant allow profile mds
2015-11-17 12:47:38.000392 7ffa8b2e7700 12 mon.000-s-ragnarok@0(leader).mds e9429 preprocess_beacon mdsbeacon(1614101/000-s-ragnarok up:active seq 203 v9429) v4 from mds.0 xxx.xxx.xxx.xxx:6800/2411 compat={},rocom
pat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table}
2015-11-17 12:47:38.000407 7ffa8b2e7700 15 mon.000-s-ragnarok@0(leader).mds e9429 _note_beacon mdsbeacon(1614101/000-s-ragnarok up:active seq 203 v9429) v4 noting time
2015-11-17 12:47:38.000416 7ffa8b2e7700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b5350f40 0x5602b51a0100 mdsbeacon(1614101/000-s-ragnarok up:active seq 203 v9429) v4
2015-11-17 12:47:42.000556 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1 _ms_dispatch existing session 0x5602b5178680 for mds.? xxx.xxx.xxx.xxx:6800/2411
2015-11-17 12:47:42.000569 7ffa8b2e7700 20 mon.000-s-ragnarok@0(leader) e1  caps allow profile mds
2015-11-17 12:47:42.000594 7ffa8b2e7700 10 mon.000-s-ragnarok@0(leader).mds e9429 preprocess_query mdsbeacon(1614101/000-s-ragnarok up:active seq 204 v9429) v4 from mds.0 xxx.xxx.xxx.xxx:6800/2411
2015-11-17 12:47:42.000606 7ffa8b2e7700 20 is_capable service=mds command= exec on cap allow profile mds
2015-11-17 12:47:42.000610 7ffa8b2e7700 20  allow so far , doing grant allow profile mds
2015-11-17 12:47:42.000616 7ffa8b2e7700 12 mon.000-s-ragnarok@0(leader).mds e9429 preprocess_beacon mdsbeacon(1614101/000-s-ragnarok up:active seq 204 v9429) v4 from mds.0 xxx.xxx.xxx.xxx:6800/2411 compat={},rocom
pat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table}
2015-11-17 12:47:42.000630 7ffa8b2e7700 15 mon.000-s-ragnarok@0(leader).mds e9429 _note_beacon mdsbeacon(1614101/000-s-ragnarok up:active seq 204 v9429) v4 noting time
2015-11-17 12:47:42.000640 7ffa8b2e7700  2 mon.000-s-ragnarok@0(leader) e1 send_reply 0x5602b5350760 0x5602b519e600 mdsbeacon(1614101/000-s-ragnarok up:active seq 204 v9429) v4
2015-11-17 12:47:42.905260 7ffa8bae8700 10 mon.000-s-ragnarok@0(leader).mds e9429 e9429: 1/1/0 up {0=000-s-ragnarok=up:active}

Se

--
 Mykola 
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]

  Powered by Linux