It honestly just looks like your MDS and MGR daemons are not configured
to start automatically. Try starting them manually and then if that
fixes the things, go through and enable them to start automatically.
Assuming you use systemctl the commands to check and fix this would be
something like these. The first one will show you all of the things
started with ceph.target.
sudo systemctl list-dependencies ceph.target
sudo systemctl enable ceph-mgr@servername
sudo systemctl enable ceph-mds@servername
On Mon, Dec 11, 2017 at 1:08 PM Jan Pekař - Imatic <jan.pekar@xxxxxxxxx
<mailto:jan.pekar@xxxxxxxxx>> wrote:
Hi all,
hope that somebody can help me. I have home ceph installation.
After power failure (it can happen in datacenter also) my ceph booted in
non-consistent state.
I was backfilling data on one new disk during power failure. First time
it booted without some OSDs, but I fixed that. Now I have all my OSD's
running, but cluster state looks like this after some time. :
cluster:
id: 2d9bf17f-3d50-4a59-8359-abc8328fe801
health: HEALTH_WARN
1 filesystem is degraded
1 filesystem has a failed mds daemon
noout,nodeep-scrub flag(s) set
no active mgr
317162/12520262 objects misplaced (2.533%)
Reduced data availability: 52 pgs inactive, 29 pgs down, 1
pg peering, 1 pg stale
Degraded data redundancy: 2099528/12520262 objects
degraded
(16.769%), 427 pgs unclean, 368 pgs degraded, 368 pgs undersized
1/3 mons down, quorum imatic-mce-2,imatic-mce
services:
mon: 3 daemons, quorum imatic-mce-2,imatic-mce, out of quorum:
obyvak
mgr: no daemons active
mds: cephfs-0/1/1 up , 1 failed
osd: 8 osds: 8 up, 8 in; 61 remapped pgs
flags noout,nodeep-scrub
data:
pools: 8 pools, 896 pgs
objects: 4446k objects, 9119 GB
usage: 9698 GB used, 2290 GB / 11988 GB avail
pgs: 2.455% pgs unknown
3.348% pgs not active
2099528/12520262 objects degraded (16.769%)
317162/12520262 objects misplaced (2.533%)
371 stale+active+clean
183 active+undersized+degraded
154 stale+active+undersized+degraded
85 active+clean
22 unknown
19 stale+down
14
stale+active+undersized+degraded+remapped+backfill_wait
13 active+undersized+degraded+remapped+backfill_wait
10 down
6 active+clean+remapped
6 stale+active+clean+remapped
5 stale+active+remapped+backfill_wait
2 active+remapped+backfill_wait
2 stale+active+undersized+degraded+remapped+backfilling
1 active+undersized+degraded+remapped
1 active+undersized+degraded+remapped+backfilling
1 stale+peering
1 stale+active+clean+scrubbing
There are all OSD's up and running. Before that I completed
ceph osd out
on one of my disk and removed that disk from cluster because I don't
want to use it anymore. It triggered crush reweight and started to
rebuild my date. I thinkg that should not put my data in danger even I
saw that some of my PG's were undersized (why?) - but it is not now the
think.
When I try to do
ceph pg dump
I have no response.
But ceph osd dump show weird number of osd's on temporary PG's like
number 2147483647 <tel:(214)%20748-3647>. I thing that there is some
problem in some mon or
other database and peering process cannot complete.
What can I do next? I believed that cluster so much, so I have some data
I want back. Thank you very much. for help.
My ceph osd dump looks like this:
epoch 29442
fsid 2d9bf17f-3d50-4a59-8359-abc8328fe801
created 2014-12-10 23:00:49.140787
modified 2017-12-11 18:54:01.134091
flags noout,nodeep-scrub,sortbitwise,recovery_deletes
crush_version 14
full_ratio 0.97
backfillfull_ratio 0.91
nearfull_ratio 0.9
require_min_compat_client firefly
min_compat_client firefly
require_osd_release luminous
pool 0 'data' replicated size 2 min_size 1 crush_rule 0 object_hash
rjenkins pg_num 64 pgp_num 64 last_change 27537 flags hashpspool
crash_replay_interval 45 min_read_recency_for_promote 1
min_write_recency_for_promote 1 stripe_width 0 application cephfs
pool 1 'metadata' replicated size 3 min_size 1 crush_rule 1 object_hash
rjenkins pg_num 64 pgp_num 64 last_change 27537 flags hashpspool
min_read_recency_for_promote 1 min_write_recency_for_promote 1
stripe_width 0 application cephfs
pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash
rjenkins pg_num 64 pgp_num 64 last_change 28088 flags hashpspool
min_read_recency_for_promote 1 min_write_recency_for_promote 1
stripe_width 0 application rbd
removed_snaps [1~5]
pool 3 'nonreplicated' replicated size 1 min_size 1 crush_rule 2
object_hash rjenkins pg_num 192 pgp_num 192 last_change 27537 flags
hashpspool min_read_recency_for_promote 1 min_write_recency_for_promote
1 stripe_width 0 application cephfs
pool 4 'replicated' replicated size 2 min_size 1 crush_rule 0
object_hash rjenkins pg_num 192 pgp_num 192 last_change 27537 lfor
17097/17097 flags hashpspool min_read_recency_for_promote 1
min_write_recency_for_promote 1 stripe_width 0 application cephfs
pool 10 'erasure_3_1' erasure size 4 min_size 3 crush_rule 3 object_hash
rjenkins pg_num 128 pgp_num 128 last_change 27537 lfor 9127/9127 flags
hashpspool tiers 11 read_tier 11 write_tier 11
min_write_recency_for_promote 1 stripe_width 4128 application cephfs
pool 11 'erasure_3_1_hot' replicated size 2 min_size 1 crush_rule 1
object_hash rjenkins pg_num 128 pgp_num 128 last_change 9910 flags
hashpspool,incomplete_clones tier_of 10 cache_mode writeback
target_bytes 5368709120 hit_set bloom{false_positive_probability: 0.05,
target_size: 0, seed: 0} 0s x0 decay_rate 0 search_last_n 1
min_write_recency_for_promote 1 stripe_width 0
pool 12 'test' replicated size 1 min_size 1 crush_rule 4 object_hash
rjenkins pg_num 64 pgp_num 64 last_change 27463 flags hashpspool
stripe_width 0
max_osd 8
osd.0 up in weight 1 up_from 29416 up_thru 29433 down_at 29407
last_clean_interval [29389,29406) 192.168.11.165:6800/9273
<http://192.168.11.165:6800/9273>
192.168.11.165:6801/9273 <http://192.168.11.165:6801/9273>
192.168.11.165:6802/9273 <http://192.168.11.165:6802/9273>
192.168.11.165:6803/9273 <http://192.168.11.165:6803/9273> exists,up
630fe0dc-9ec0-456a-bf15-51d6d3ba462d
osd.1 up in weight 1 up_from 29422 up_thru 29437 down_at 29407
last_clean_interval [29390,29406) 192.168.11.165:6816/9336
<http://192.168.11.165:6816/9336>
192.168.11.165:6817/9336 <http://192.168.11.165:6817/9336>
192.168.11.165:6818/9336 <http://192.168.11.165:6818/9336>
192.168.11.165:6819/9336 <http://192.168.11.165:6819/9336> exists,up
ef583c8d-171f-47c4-8a9e-e9eb913cb272
osd.2 up in weight 1 up_from 29409 up_thru 29433 down_at 29407
last_clean_interval [29389,29406) 192.168.11.165:6804/9285
<http://192.168.11.165:6804/9285>
192.168.11.165:6805/9285 <http://192.168.11.165:6805/9285>
192.168.11.165:6806/9285 <http://192.168.11.165:6806/9285>
192.168.11.165:6807/9285 <http://192.168.11.165:6807/9285> exists,up
1de26ef5-319d-426e-ad75-65aedbbd0328
osd.3 up in weight 1 up_from 29430 up_thru 29439 down_at 29410
last_clean_interval [29391,29406) 192.168.11.165:6824/12146
<http://192.168.11.165:6824/12146>
192.168.11.165:6825/12146 <http://192.168.11.165:6825/12146>
192.168.11.165:6826/12146 <http://192.168.11.165:6826/12146>
192.168.11.165:6827/12146 <http://192.168.11.165:6827/12146>
exists,up 5b63a084-cb0c-4e6a-89c1-9d2fc70cea02
osd.4 up in weight 1 up_from 29442 up_thru 0 down_at 29347
last_clean_interval [29317,29343) 192.168.11.165:6828/15193
<http://192.168.11.165:6828/15193>
192.168.11.165:6829/15193 <http://192.168.11.165:6829/15193>
192.168.11.165:6830/15193 <http://192.168.11.165:6830/15193>
192.168.11.165:6831/15193 <http://192.168.11.165:6831/15193>
exists,up ee9d758d-f2df-41b6-9320-ce89f54c116b
osd.5 up in weight 1 up_from 29414 up_thru 29431 down_at 29413
last_clean_interval [29390,29406) 192.168.11.165:6812/9321
<http://192.168.11.165:6812/9321>
192.168.11.165:6813/9321 <http://192.168.11.165:6813/9321>
192.168.11.165:6814/9321 <http://192.168.11.165:6814/9321>
192.168.11.165:6815/9321 <http://192.168.11.165:6815/9321> exists,up
d1077d42-2c92-4afd-a11e-02fdd59b393b
osd.6 up in weight 1 up_from 29413 up_thru 29433 down_at 29407
last_clean_interval [29390,29406) 192.168.11.165:6820/9345
<http://192.168.11.165:6820/9345>
192.168.11.165:6821/9345 <http://192.168.11.165:6821/9345>
192.168.11.165:6822/9345 <http://192.168.11.165:6822/9345>
192.168.11.165:6823/9345 <http://192.168.11.165:6823/9345> exists,up
f55da9e5-0c03-43fa-af59-56add845c706
osd.7 up in weight 1 up_from 29422 up_thru 29433 down_at 29407
last_clean_interval [29389,29406) 192.168.11.165:6808/9309
<http://192.168.11.165:6808/9309>
192.168.11.165:6809/9309 <http://192.168.11.165:6809/9309>
192.168.11.165:6810/9309 <http://192.168.11.165:6810/9309>
192.168.11.165:6811/9309 <http://192.168.11.165:6811/9309> exists,up
1e75647b-a1fc-4672-957f-ce5c2b0f4a43
pg_temp 0.0 [0,5]
pg_temp 0.2c [0,6]
pg_temp 0.31 [7,5]
pg_temp 0.33 [0,2]
pg_temp 1.3 [1,6,5]
pg_temp 1.1d [1,3,2]
pg_temp 1.27 [6,5,2]
pg_temp 1.2c [3,1,5]
pg_temp 1.36 [6,3,2]
pg_temp 1.37 [6,1,5]
pg_temp 1.38 [6,3,1]
pg_temp 1.3f [3,5,1]
pg_temp 4.0 [3,6]
pg_temp 4.4 [0,6]
pg_temp 4.9 [0,7]
pg_temp 4.13 [6,3]
pg_temp 4.1a [6,5]
pg_temp 4.41 [6,2]
pg_temp 4.4b [0,1]
pg_temp 4.5c [6,0]
pg_temp 4.76 [0,6]
pg_temp 4.87 [0,7]
pg_temp 4.9d [0,3]
pg_temp 4.a9 [6,7]
pg_temp 10.1 [0,7,2,1]
pg_temp 10.2 [3,0,1,7]
pg_temp 10.5 [2,6,3,0]
pg_temp 10.7 [6,7,0,3]
pg_temp 10.a [0,3,5,6]
pg_temp 10.c [7,6,5,0]
pg_temp 10.f [7,3,6,0]
pg_temp 10.1c [6,1,0,3]
pg_temp 10.25 [0,7,5,3]
pg_temp 10.26 [3,2,5,1]
pg_temp 10.29 [7,5,2,0]
pg_temp 10.2f [1,5,7,0]
pg_temp 10.3b [7,3,0,6]
pg_temp 10.41 [0,1,3,7]
pg_temp 10.47 [3,0,7,5]
pg_temp 10.4c [7,3,5,1]
pg_temp 10.51 [1,7,0,2]
pg_temp 10.54 [3,0,7,5]
pg_temp 10.55 [7,1,0,3]
pg_temp 10.5a [7,0,3,2]
pg_temp 10.5b [2147483647 <tel:(214)%20748-3647>,0,5,6]
pg_temp 10.5e [7,2,3,0]
pg_temp 10.5f [6,3,2,1]
pg_temp 10.63 [7,5,3,2]
pg_temp 10.64 [3,6,7,1]
pg_temp 10.66 [0,1,5,7]
pg_temp 10.6c [7,3,2,0]
pg_temp 10.6f [6,3,2,7]
pg_temp 10.70 [0,7,6,1]
pg_temp 10.72 [7,6,1,3]
pg_temp 10.73 [6,2147483647 <tel:(214)%20748-3647>,7,0]
pg_temp 10.74 [7,1,6,3]
pg_temp 10.7c [0,6,1,7]
pg_temp 10.7f [0,7,1,2]
pg_temp 11.27 [6,5]
pg_temp 11.38 [3,6]
pg_temp 11.78 [3,1]
Thank you all for help. It is important for me.
Jan Pekar
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
