One of the issues was a different version of ceph on the nodes. They are now all back to version 9.0.2 and things are looking a bit better.
We've had some help from a ceph engineer, and the OSD's are now all up and running:
[root@stor2 ceph]# ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-13 1.00000 root spare
-9 1.00000 host stor4-spare
4 1.00000 osd.4 up 1.00000 1.00000
-12 2.00000 root meta
-7 1.00000 host stor2-meta
7 1.00000 osd.7 up 1.00000 1.00000
-8 1.00000 host stor4-meta
5 1.00000 osd.5 up 1.00000 1.00000
-11 2.00000 root ssd
-4 1.00000 host stor2-ssd
2 1.00000 osd.2 up 1.00000 1.00000
-5 1.00000 host stor4-ssd
3 1.00000 osd.3 up 1.00000 1.00000
-10 291.03998 root default
-2 72.75999 host stor2
0 72.75999 osd.0 up 1.00000 1.00000
-3 72.75999 host stor4
1 72.75999 osd.1 up 1.00000 1.00000
-1 72.75999 host stor1
6 72.75999 osd.6 up 1.00000 1.00000
-14 72.75999 host stor3
8 72.75999 osd.8 up 1.00000 1.00000
Here's an updated ceph status:
[root@stor2 ceph]# ceph -s
cluster 926a1578-48e8-4559-b1b5-1f1a9b8e7566
health HEALTH_ERR
mds rank 2 is damaged
mds cluster is degraded
mds stor4 is laggy
monmap e23: 1 mons at {0=192.168.202.101:6789/0}
election epoch 1, quorum 0 0
mdsmap e11400: 1/3/2 up {0=stor4=up:replay(laggy or crashed)}, 1 damaged
osdmap e12828: 9 osds: 9 up, 9 in
pgmap v22081981: 224 pgs, 4 pools, 101 TB data, 62358 kobjects
203 TB used, 90068 GB / 291 TB avail
222 active+clean
2 active+clean+scrubbing+deep
And here's a ceph mds dump:
[root@stor2 ceph]# ceph mds dump
dumped mdsmap epoch 11400
epoch 11400
flags 0
created 2015-04-20 02:39:19.238757
modified 2016-05-14 17:15:24.482294
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
last_failure 11399
last_failure_osd_epoch 12827
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table}
max_mds 2
in 0,1,2
up {0=10024473}
failed
damaged 2
stopped
data_pools 1
metadata_pool 2
inline_data disabled
10024473: 192.168.202.13:6816/3461545 'stor4' mds.0.276 up:replay seq 1 laggy since 2016-05-14 17:15:24.482177
But as you can see above, the mds is still damaged and the cluster is degraded, and our files are still inaccessible.
Any ideas how to fix an mds with damaged rank?
Thanks again,
Skaag
> On May 14, 2016, at 9:00 AM, Skaag Argonius <skaag@xxxxxxxxx> wrote:
>
> We have a rather urgent situation, and I need help doing one of two things:
>
> 1. Fix the MDS and regain a working cluster (ideal)
> 2. Find a way to extract the contents so I can move it to a new cluster (need to do this anyway)
>
> We have 4 physical storage machines: stor1, stor2, stor3, stor4
> The MDS is on stor2.
>
> I'm including as much info as I can below.
>
>
> -------8<-----------------------------------------------------------------------------
>
> Cluster status:
>
> [root@stor1 /]# ceph -s
> cluster 926a1578-48e8-4559-b1b5-1f1a9b8e7566
> health HEALTH_ERR
> 2 pgs backfill
> 24 pgs backfilling
> 122 pgs degraded
> 14 pgs stale
> 122 pgs stuck degraded
> 14 pgs stuck stale
> 122 pgs stuck unclean
> 122 pgs stuck undersized
> 122 pgs undersized
> recovery 26146044/127710264 objects degraded (20.473%)
> recovery 21267236/127710264 objects misplaced (16.653%)
> mds ranks 0,2 have failed
> mds cluster is degraded
> mds gentoo-backup is laggy
> monmap e15: 1 mons at {0=192.168.202.101:6789/0}
> election epoch 1, quorum 0 0
> mdsmap e11373: 1/2/2 up {1=gentoo-backup=up:replay(laggy or crashed)}, 2 failed
> osdmap e12755: 9 osds: 6 up, 6 in; 26 remapped pgs
> pgmap v22065162: 224 pgs, 4 pools, 101 TB data, 62358 kobjects
> 169 TB used, 50604 GB / 218 TB avail
> 26146044/127710264 objects degraded (20.473%)
> 21267236/127710264 objects misplaced (16.653%)
> 96 active+undersized+degraded
> 87 active+clean
> 24 active+undersized+degraded+remapped+backfilling
> 14 stale+active+clean
> 2 active+undersized+degraded+remapped+wait_backfill
> 1 active+clean+scrubbing+deep
> recovery io 100993 kB/s, 68 objects/s
>
> -------8<-----------------------------------------------------------------------------
>
>
> History:
>
> We had a single MDS, things were working fine. One of our sysadmins changed permissions in some sensitive folder, apparently (I don't have all the details).
>
> As a result the MDS went down. He then tried to bring a backup mds into the mix (called 'gentoo-backup'), but I think that MDS was an exact replica of the first MDS, so obviously it wasn't going to work well...
>
> We tried to kick the backup MDS out, tried to mark it as failed, tried to make it inactive, tried to reduce max_mds to 1, nothing helped.
>
>
> -------8<-----------------------------------------------------------------------------
>
> MDS Crash logs:
>
> 2016-05-13 15:19:27.447753 7f7a82580700 -1 log_channel(cluster) log [ERR] : corrupt journal event at 4194304~472 / 10426264
> 2016-05-13 15:19:27.448901 7f7a82580700 -1 mds/Beacon.cc: In function 'void Beacon::notify_health(const MDS*)' thread 7f7a82580700 time 2016-05-13 15:19:27.447829
> mds/Beacon.cc: 291: FAILED assert(mds->mds_lock.is_locked_by_me())
>
> ceph version 9.0.2 (be422c8f5b494c77ebcf0f7b95e5d728ecacb7f0)
> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x98b645]
> 2: (Beacon::notify_health(MDS const*)+0x74) [0x5d7d54]
> 3: (MDS::damaged()+0x42) [0x5a9bd2]
> 4: (MDLog::_replay_thread()+0x1c02) [0x803e62]
> 5: (MDLog::ReplayThread::entry()+0xd) [0x5c88bd]
> 6: (()+0x7df5) [0x7f7a8c564df5]
> 7: (clone()+0x6d) [0x7f7a8b24b1ad]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>
> -------8<-----------------------------------------------------------------------------
>
> More from MDS log:
>
> 2016-05-13 15:19:27.451896 7f7a82580700 -1 *** Caught signal (Aborted) **
> in thread 7f7a82580700
>
> ceph version 9.0.2 (be422c8f5b494c77ebcf0f7b95e5d728ecacb7f0)
> 1: /usr/bin/ceph-mds() [0x8916b2]
> 2: (()+0xf130) [0x7f7a8c56c130]
> 3: (gsignal()+0x37) [0x7f7a8b18a5d7]
> 4: (abort()+0x148) [0x7f7a8b18bcc8]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f7a8ba8e9b5]
> 6: (()+0x5e926) [0x7f7a8ba8c926]
> 7: (()+0x5e953) [0x7f7a8ba8c953]
> 8: (()+0x5eb73) [0x7f7a8ba8cb73]
> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0x98b83a]
> 10: (Beacon::notify_health(MDS const*)+0x74) [0x5d7d54]
> 11: (MDS::damaged()+0x42) [0x5a9bd2]
> 12: (MDLog::_replay_thread()+0x1c02) [0x803e62]
> 13: (MDLog::ReplayThread::entry()+0xd) [0x5c88bd]
> 14: (()+0x7df5) [0x7f7a8c564df5]
> 15: (clone()+0x6d) [0x7f7a8b24b1ad]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>
> ...
> ...
> ...
>
> log_file /var/log/ceph/ceph-mds.stor2.log
> --- end dump of recent events ---
> 2016-05-13 15:54:53.300449 7f1e1f47c7c0 0 ceph version 9.0.2 (be422c8f5b494c77ebcf0f7b95e5d728ecacb7f0), process ceph-mds, pid 3935832
> 2016-05-13 15:54:53.324083 7f1e1f47c7c0 -1 mds.-1.0 log_to_monitors {default=true}
> 2016-05-13 15:54:53.572405 7f1e19aaf700 1 mds.-1.0 handle_mds_map standby
> 2016-05-13 15:56:17.247033 7f1e151a5700 -1 mds.-1.0 *** got signal Terminated ***
> 2016-05-13 15:56:17.250635 7f1e151a5700 1 mds.-1.0 suicide. wanted down:dne, now up:standby
> 2016-05-13 15:56:18.405716 7f6ba594c7c0 0 ceph version 9.0.2 (be422c8f5b494c77ebcf0f7b95e5d728ecacb7f0), process ceph-mds, pid 3940158
> 2016-05-13 15:56:18.428399 7f6ba594c7c0 -1 mds.-1.0 log_to_monitors {default=true}
> 2016-05-13 15:56:18.984429 7f6b9ff7f700 1 mds.-1.0 handle_mds_map standby
> 2016-05-13 15:56:59.516357 7f6b9ff7f700 1 mds.-1.0 handle_mds_map standby
> 2016-05-13 15:57:02.637735 7f6b9ff7f700 1 mds.-1.0 handle_mds_map standby
> 2016-05-13 15:57:05.774427 7f6b9ff7f700 1 mds.-1.0 handle_mds_map standby
>
> -------8<-----------------------------------------------------------------------------
>
> Last portion from MDS log on stor2 (from right now):
>
> 2016-05-13 19:10:15.609077 7f0e77eab780 0 ceph version 9.1.0 (3be81ae6cf17fcf689cd6f187c4615249fea4f61), process ceph-mds, pid 4028429
> 2016-05-13 19:10:16.464975 7f0e726d8700 1 mds.stor2 handle_mds_map standby
> 2016-05-13 19:10:16.467750 7f0e726d8700 1 mds.0.267 handle_mds_map i am now mds.0.267
> 2016-05-13 19:10:16.467757 7f0e726d8700 1 mds.0.267 handle_mds_map state change up:boot --> up:replay
> 2016-05-13 19:10:16.467778 7f0e726d8700 1 mds.0.267 replay_start
> 2016-05-13 19:10:16.467784 7f0e726d8700 1 mds.0.267 recovery set is 1
> 2016-05-13 19:10:16.467789 7f0e726d8700 1 mds.0.267 waiting for osdmap 12618 (which blacklists prior instance)
> 2016-05-13 19:10:16.470750 7f0e6d4cc700 0 mds.0.cache creating system inode with ino:100
> 2016-05-13 19:10:16.470880 7f0e6d4cc700 0 mds.0.cache creating system inode with ino:1
> 2016-05-13 19:10:16.878435 7f0e6c1c7700 1 mds.0.267 replay_done
> 2016-05-13 19:10:16.878454 7f0e6c1c7700 1 mds.0.267 making mds journal writeable
> 2016-05-13 19:10:17.681414 7f0e726d8700 1 mds.0.267 handle_mds_map i am now mds.0.267
> 2016-05-13 19:10:17.681419 7f0e726d8700 1 mds.0.267 handle_mds_map state change up:replay --> up:resolve
> 2016-05-13 19:10:17.681431 7f0e726d8700 1 mds.0.267 resolve_start
> 2016-05-13 19:10:17.681432 7f0e726d8700 1 mds.0.267 reopen_log
> 2016-05-13 19:11:45.822580 7f0e726d8700 1 mds.0.cache handle_mds_failure mds.1 : recovery peers are 1
> 2016-05-13 19:12:17.462003 7f0e726d8700 1 mds.0.cache handle_mds_failure mds.1 : recovery peers are 1
> 2016-05-13 19:12:31.705710 7f0e726d8700 1 mds.0.cache handle_mds_failure mds.1 : recovery peers are 1
> 2016-05-13 19:12:51.942145 7f0e726d8700 1 mds.0.cache handle_mds_failure mds.1 : recovery peers are 1
> 2016-05-13 19:13:59.720073 7f0e6edd0700 -1 mds.stor2 *** got signal Terminated ***
> 2016-05-13 19:13:59.720130 7f0e6edd0700 1 mds.stor2 suicide. wanted state up:resolve
> 2016-05-13 19:13:59.722837 7f0e6edd0700 1 mds.0.267 shutdown: shutting down rank 0
>
>
> -------8<-----------------------------------------------------------------------------
>
> My own experience with storage systems is limited to NFS, GlusterFS and various exotic Fuse based filesystems. I came into this startup with ceph already installed the way it is (I inherited it), so be gentle with me please :-)
>
> Thanks in advance for any help/advice you can give!
>
> Skaag
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
