Yes, ceph pg <PGID> query, not dump. Sorry about that. Are you having problems with OSD stability? There's a lot of history in the [recovery_state][past_intervals]. That's normal when OSDs go down, and out, and come back up and in. You have a lot of history there. You might even be getting into the point that you have so much failover history, the OSDs can't process it all before they hit the suicide timeout. [recovery_state][probing_osds] lists a lot of OSDs that have recently owned these PGs. If the OSDs are crashing frequently, you need to get that under control before proceeding. Once the OSDs are stable, I think Ceph just needs to scrub and deep-scrub those PGs. Until Ceph clears out the [recovery_state][probing_osds] section in the pg query, it's not going to do anything. ceph osd lost hears you, but doesn't trust you. Ceph won't do anything until it's actually checked those OSDs itself. Scrubbing and Deep scrubbing should convince it. Once that [recovery_state][probing_osds] section is gone, you should see the [recovery_state][past_intervals] section shrink or disappear. I don't have either section in my pg query. Once that happens, your ceph pg repair or ceph pg force_create_pg should finally have some effect. You may or may not need to re-issue those commands. On Tue, Aug 12, 2014 at 9:32 PM, Riederer, Michael <Michael.Riederer at br.de> wrote: > Hi Craig, > > # ceph pg 2.587 query > # ceph pg 2.c1 query > # ceph pg 2.92 query > # ceph pg 2.e3 query > > Please download the output form here: > http://server.riederer.org/ceph-user/ > > ##################################### > > > It is not possible to map a rbd: > > # rbd map testshareone --pool rbd --name client.admin > rbd: add failed: (5) Input/output error > > I found that: > http://permalink.gmane.org/gmane.comp.file-systems.ceph.user/11405 > # ceph osd getcrushmap -o crushmap.bin > got crush map from osdmap epoch 3741 > # crushtool -i crushmap.bin --set-chooseleaf_vary_r 0 -o crushmap-new.bin > # ceph osd setcrushmap -i crushmap-new.bin > set crush map > > The Cluster had to do some. Now it looks a bit different. > > It is still not possible to map a rbd. > > root at ceph-admin-storage:~# ceph -s > cluster 6b481875-8be5-4508-b075-e1f660fd7b33 > health HEALTH_WARN 4 pgs incomplete; 4 pgs stuck inactive; 4 pgs > stuck unclean > monmap e2: 3 mons at {ceph-1-storage= > 10.65.150.101:6789/0,ceph-2-storage=10.65.150.102:6789/0,ceph-3-storage=10.65.150.103:6789/0}, > election epoch 5010, quorum 0,1,2 > ceph-1-storage,ceph-2-storage,ceph-3-storage > osdmap e34206: 55 osds: 55 up, 55 in > pgmap v10838368: 6144 pgs, 3 pools, 11002 GB data, 2762 kobjects > 22078 GB used, 79932 GB / 102010 GB avail > 6140 active+clean > 4 incomplete > > root at ceph-admin-storage:~# ceph health detail > HEALTH_WARN 4 pgs incomplete; 4 pgs stuck inactive; 4 pgs stuck unclean > pg 2.92 is stuck inactive since forever, current state incomplete, last > acting [8,13] > pg 2.c1 is stuck inactive since forever, current state incomplete, last > acting [13,8] > pg 2.e3 is stuck inactive since forever, current state incomplete, last > acting [20,8] > pg 2.587 is stuck inactive since forever, current state incomplete, last > acting [13,8] > > pg 2.92 is stuck unclean since forever, current state incomplete, last > acting [8,13] > pg 2.c1 is stuck unclean since forever, current state incomplete, last > acting [13,8] > pg 2.e3 is stuck unclean since forever, current state incomplete, last > acting [20,8] > pg 2.587 is stuck unclean since forever, current state incomplete, last > acting [13,8] > pg 2.587 is incomplete, acting [13,8] > pg 2.e3 is incomplete, acting [20,8] > pg 2.c1 is incomplete, acting [13,8] > > pg 2.92 is incomplete, acting [8,13] > > ####################################################################### > > After updating to firefly, I did the following: > > # ceph health detail > HEALTH_WARN crush map has legacy tunables crush map has legacy tunables; > see http://ceph.com/docs/master/rados/operations/crush-map/#tunables > > # ceph osd crush tunables optimal > adjusted tunables profile to optimal > > Mike > ------------------------------ > *Von:* Craig Lewis [clewis at centraldesktop.com] > *Gesendet:* Dienstag, 12. August 2014 20:02 > *An:* Riederer, Michael > *Cc:* Karan Singh; ceph-users at lists.ceph.com > > *Betreff:* Re: [ceph-users] HEALTH_WARN 4 pgs incomplete; 4 pgs stuck > inactive; 4 pgs stuck unclean > > For the incomplete PGs, can you give me the output of > ceph pg <PGID> dump > > I'm interested in the recovery_state key of that JSON data. > > > > On Tue, Aug 12, 2014 at 5:29 AM, Riederer, Michael <Michael.Riederer at br.de > > wrote: > >> Sorry, but I think that does not help me. I forgot to mention something about >> the operating system: >> >> root at ceph-1-storage:~# dpkg -l | grep libleveldb1 >> ii libleveldb1 1.12.0-1precise.ceph >> fast key-value storage library >> root at ceph-1-storage:~# lsb_release -a >> No LSB modules are available. >> Distributor ID: Ubuntu >> Description: Ubuntu 12.04.5 LTS >> Release: 12.04 >> Codename: precise >> root at ceph-1-storage:~# uname -a >> Linux ceph-1-storage 3.5.0-52-generic #79~precise1-Ubuntu SMP Fri Jul 4 >> 21:03:49 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux >> >> libleveldb1 is greater than the mentioned version 1.9.0-1 ~ bpo70 + 1. >> >> All ceph nodes are IBM x3650 with Intel Xeon CPUs 2.00 GHz and 8 GB RAM, >> ok all very old, about eight years, >> but are still running. >> >> Mike >> >> >> >> ------------------------------ >> *Von:* Karan Singh [karan.singh at csc.fi] >> *Gesendet:* Dienstag, 12. August 2014 13:00 >> >> *An:* Riederer, Michael >> *Cc:* ceph-users at lists.ceph.com >> *Betreff:* Re: [ceph-users] HEALTH_WARN 4 pgs incomplete; 4 pgs stuck >> inactive; 4 pgs stuck unclean >> >> I am not sure if this helps , but have a look >> https://www.mail-archive.com/ceph-users at lists.ceph.com/msg10078.html >> >> - Karan - >> >> On 12 Aug 2014, at 12:04, Riederer, Michael <Michael.Riederer at br.de> >> wrote: >> >> Hi Karan, >> >> root at ceph-admin-storage:~/ceph-cluster/crush-map-4-ceph-user-list# ceph >> osd getcrushmap -o crushmap.bin >> got crush map from osdmap epoch 30748 >> root at ceph-admin-storage:~/ceph-cluster/crush-map-4-ceph-user-list# >> crushtool -d crushmap.bin -o crushmap.txt >> root at ceph-admin-storage:~/ceph-cluster/crush-map-4-ceph-user-list# cat >> crushmap.txt >> # begin crush map >> tunable choose_local_tries 0 >> tunable choose_local_fallback_tries 0 >> tunable choose_total_tries 50 >> tunable chooseleaf_descend_once 1 >> tunable chooseleaf_vary_r 1 >> >> # devices >> device 0 osd.0 >> device 1 osd.1 >> device 2 osd.2 >> device 3 osd.3 >> device 4 osd.4 >> device 5 osd.5 >> device 6 osd.6 >> device 7 osd.7 >> device 8 osd.8 >> device 9 osd.9 >> device 10 osd.10 >> device 11 osd.11 >> device 12 osd.12 >> device 13 osd.13 >> device 14 osd.14 >> device 15 osd.15 >> device 16 osd.16 >> device 17 osd.17 >> device 18 osd.18 >> device 19 osd.19 >> device 20 osd.20 >> device 21 device21 >> device 22 osd.22 >> device 23 osd.23 >> device 24 osd.24 >> device 25 osd.25 >> device 26 osd.26 >> device 27 device27 >> device 28 osd.28 >> device 29 osd.29 >> device 30 osd.30 >> device 31 osd.31 >> device 32 osd.32 >> device 33 osd.33 >> device 34 osd.34 >> device 35 osd.35 >> device 36 osd.36 >> device 37 osd.37 >> device 38 osd.38 >> device 39 osd.39 >> device 40 device40 >> device 41 device41 >> device 42 osd.42 >> device 43 osd.43 >> device 44 osd.44 >> device 45 osd.45 >> device 46 osd.46 >> device 47 osd.47 >> device 48 osd.48 >> device 49 osd.49 >> device 50 osd.50 >> device 51 osd.51 >> device 52 osd.52 >> device 53 osd.53 >> device 54 osd.54 >> device 55 osd.55 >> device 56 osd.56 >> device 57 osd.57 >> device 58 osd.58 >> >> # types >> type 0 osd >> type 1 host >> type 2 rack >> type 3 row >> type 4 room >> type 5 datacenter >> type 6 root >> >> # buckets >> host ceph-1-storage { >> id -2 # do not change unnecessarily >> # weight 19.330 >> alg straw >> hash 0 # rjenkins1 >> item osd.0 weight 0.910 >> item osd.2 weight 0.910 >> item osd.3 weight 0.910 >> item osd.4 weight 1.820 >> item osd.9 weight 1.360 >> item osd.11 weight 0.680 >> item osd.6 weight 3.640 >> item osd.5 weight 1.820 >> item osd.7 weight 3.640 >> item osd.8 weight 3.640 >> } >> host ceph-2-storage { >> id -3 # do not change unnecessarily >> # weight 20.000 >> alg straw >> hash 0 # rjenkins1 >> item osd.14 weight 3.640 >> item osd.18 weight 1.360 >> item osd.19 weight 1.360 >> item osd.15 weight 3.640 >> item osd.1 weight 3.640 >> item osd.12 weight 3.640 >> item osd.22 weight 0.680 >> item osd.23 weight 0.680 >> item osd.26 weight 0.680 >> item osd.36 weight 0.680 >> } >> host ceph-5-storage { >> id -4 # do not change unnecessarily >> # weight 11.730 >> alg straw >> hash 0 # rjenkins1 >> item osd.32 weight 0.270 >> item osd.37 weight 0.270 >> item osd.42 weight 0.270 >> item osd.43 weight 1.820 >> item osd.44 weight 1.820 >> item osd.45 weight 1.820 >> item osd.46 weight 1.820 >> item osd.47 weight 1.820 >> item osd.48 weight 1.820 >> } >> room room0 { >> id -8 # do not change unnecessarily >> # weight 51.060 >> alg straw >> hash 0 # rjenkins1 >> item ceph-1-storage weight 19.330 >> item ceph-2-storage weight 20.000 >> item ceph-5-storage weight 11.730 >> } >> host ceph-3-storage { >> id -5 # do not change unnecessarily >> # weight 15.920 >> alg straw >> hash 0 # rjenkins1 >> item osd.24 weight 1.820 >> item osd.25 weight 1.820 >> item osd.29 weight 1.360 >> item osd.10 weight 3.640 >> item osd.13 weight 3.640 >> item osd.20 weight 3.640 >> } >> host ceph-4-storage { >> id -6 # do not change unnecessarily >> # weight 20.000 >> alg straw >> hash 0 # rjenkins1 >> item osd.34 weight 3.640 >> item osd.38 weight 1.360 >> item osd.39 weight 1.360 >> item osd.16 weight 3.640 >> item osd.30 weight 0.680 >> item osd.35 weight 3.640 >> item osd.17 weight 3.640 >> item osd.28 weight 0.680 >> item osd.31 weight 0.680 >> item osd.33 weight 0.680 >> } >> host ceph-6-storage { >> id -7 # do not change unnecessarily >> # weight 12.720 >> alg straw >> hash 0 # rjenkins1 >> item osd.49 weight 0.450 >> item osd.50 weight 0.450 >> item osd.51 weight 0.450 >> item osd.52 weight 0.450 >> item osd.53 weight 1.820 >> item osd.54 weight 1.820 >> item osd.55 weight 1.820 >> item osd.56 weight 1.820 >> item osd.57 weight 1.820 >> item osd.58 weight 1.820 >> } >> room room1 { >> id -9 # do not change unnecessarily >> # weight 48.640 >> alg straw >> hash 0 # rjenkins1 >> item ceph-3-storage weight 15.920 >> item ceph-4-storage weight 20.000 >> item ceph-6-storage weight 12.720 >> } >> root default { >> id -1 # do not change unnecessarily >> # weight 99.700 >> alg straw >> hash 0 # rjenkins1 >> item room0 weight 51.060 >> item room1 weight 48.640 >> } >> >> # rules >> rule data { >> ruleset 0 >> type replicated >> min_size 1 >> max_size 10 >> step take default >> step chooseleaf firstn 0 type host >> step emit >> } >> rule metadata { >> ruleset 1 >> type replicated >> min_size 1 >> max_size 10 >> step take default >> step chooseleaf firstn 0 type host >> step emit >> } >> rule rbd { >> ruleset 2 >> type replicated >> min_size 1 >> max_size 10 >> step take default >> step chooseleaf firstn 0 type host >> step emit >> } >> >> # end crush map >> >> root at ceph-admin-storage:~# ceph osd dump | grep -i pool >> pool 0 'data' replicated size 2 min_size 1 crush_ruleset 0 object_hash >> rjenkins pg_num 2048 pgp_num 2048 last_change 4623 crash_replay_interval 45 >> stripe_width 0 >> pool 1 'metadata' replicated size 2 min_size 1 crush_ruleset 1 >> object_hash rjenkins pg_num 2048 pgp_num 2048 last_change 4627 stripe_width >> 0 >> pool 2 'rbd' replicated size 2 min_size 1 crush_ruleset 2 object_hash >> rjenkins pg_num 2048 pgp_num 2048 last_change 4632 stripe_width 0 >> >> >> Mike >> ------------------------------ >> *Von:* Karan Singh [karan.singh at csc.fi] >> *Gesendet:* Dienstag, 12. August 2014 10:35 >> *An:* Riederer, Michael >> *Cc:* ceph-users at lists.ceph.com >> *Betreff:* Re: [ceph-users] HEALTH_WARN 4 pgs incomplete; 4 pgs stuck >> inactive; 4 pgs stuck unclean >> >> Can you provide your cluster?s ceph osd dump | grep -i pool and >> crush map output. >> >> >> - Karan - >> >> On 12 Aug 2014, at 10:40, Riederer, Michael <Michael.Riederer at br.de> >> wrote: >> >> Hi all, >> >> How do I get my Ceph Cluster back to a healthy state? >> >> root at ceph-admin-storage:~# ceph -v >> ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6) >> root at ceph-admin-storage:~# ceph -s >> cluster 6b481875-8be5-4508-b075-e1f660fd7b33 >> health HEALTH_WARN 4 pgs incomplete; 4 pgs stuck inactive; 4 pgs >> stuck unclean >> monmap e2: 3 mons at {ceph-1-storage= >> 10.65.150.101:6789/0,ceph-2-storage=10.65.150.102:6789/0,ceph-3-storage=10.65.150.103:6789/0}, >> election epoch 5010, quorum 0,1,2 >> ceph-1-storage,ceph-2-storage,ceph-3-storage >> osdmap e30748: 55 osds: 55 up, 55 in >> pgmap v10800465: 6144 pgs, 3 pools, 11002 GB data, 2762 kobjects >> 22077 GB used, 79933 GB / 102010 GB avail >> 6138 active+clean >> 4 incomplete >> 2 active+clean+replay >> root at ceph-admin-storage:~# ceph health detail >> HEALTH_WARN 4 pgs incomplete; 4 pgs stuck inactive; 4 pgs stuck unclean >> pg 2.92 is stuck inactive since forever, current state incomplete, last >> acting [8,13] >> pg 2.c1 is stuck inactive since forever, current state incomplete, last >> acting [13,7] >> pg 2.e3 is stuck inactive since forever, current state incomplete, last >> acting [20,7] >> pg 2.587 is stuck inactive since forever, current state incomplete, last >> acting [13,5] >> pg 2.92 is stuck unclean since forever, current state incomplete, last >> acting [8,13] >> pg 2.c1 is stuck unclean since forever, current state incomplete, last >> acting [13,7] >> pg 2.e3 is stuck unclean since forever, current state incomplete, last >> acting [20,7] >> pg 2.587 is stuck unclean since forever, current state incomplete, last >> acting [13,5] >> pg 2.587 is incomplete, acting [13,5] >> pg 2.e3 is incomplete, acting [20,7] >> pg 2.c1 is incomplete, acting [13,7] >> pg 2.92 is incomplete, acting [8,13] >> root at ceph-admin-storage:~# ceph pg dump_stuck inactive >> ok >> pg_stat objects mip degr unf bytes log disklog >> state state_stamp v reported up up_primary acting >> acting_primary last_scrub scrub_stamp last_deep_scrub >> deep_scrub_stamp >> 2.92 0 0 0 0 0 0 0 incomplete 2014-08-08 >> 12:39:20.204592 0'0 30748:7729 [8,13] 8 [8,13] 8 >> 13503'1390419 2014-06-26 01:57:48.727625 13503'1390419 2014-06-22 >> 01:57:30.114186 >> 2.c1 0 0 0 0 0 0 0 incomplete 2014-08-08 >> 12:39:18.846542 0'0 30748:7117 [13,7] 13 [13,7] 13 >> 13503'1687017 2014-06-26 20:52:51.249864 13503'1687017 2014-06-22 >> 14:24:22.633554 >> 2.e3 0 0 0 0 0 0 0 incomplete 2014-08-08 >> 12:39:29.311552 0'0 30748:8027 [20,7] 20 [20,7] 20 >> 13503'1398727 2014-06-26 07:03:25.899254 13503'1398727 2014-06-21 >> 07:02:31.393053 >> 2.587 0 0 0 0 0 0 0 incomplete 2014-08-08 >> 12:39:19.715724 0'0 30748:7060 [13,5] 13 [13,5] 13 >> 13646'1542934 2014-06-26 07:48:42.089935 13646'1542934 2014-06-22 >> 07:46:20.363695 >> root at ceph-admin-storage:~# ceph osd tree >> # id weight type name up/down reweight >> -1 99.7 root default >> -8 51.06 room room0 >> -2 19.33 host ceph-1-storage >> 0 0.91 osd.0 up 1 >> 2 0.91 osd.2 up 1 >> 3 0.91 osd.3 up 1 >> 4 1.82 osd.4 up 1 >> 9 1.36 osd.9 up 1 >> 11 0.68 osd.11 up 1 >> 6 3.64 osd.6 up 1 >> 5 1.82 osd.5 up 1 >> 7 3.64 osd.7 up 1 >> 8 3.64 osd.8 up 1 >> -3 20 host ceph-2-storage >> 14 3.64 osd.14 up 1 >> 18 1.36 osd.18 up 1 >> 19 1.36 osd.19 up 1 >> 15 3.64 osd.15 up 1 >> 1 3.64 osd.1 up 1 >> 12 3.64 osd.12 up 1 >> 22 0.68 osd.22 up 1 >> 23 0.68 osd.23 up 1 >> 26 0.68 osd.26 up 1 >> 36 0.68 osd.36 up 1 >> -4 11.73 host ceph-5-storage >> 32 0.27 osd.32 up 1 >> 37 0.27 osd.37 up 1 >> 42 0.27 osd.42 up 1 >> 43 1.82 osd.43 up 1 >> 44 1.82 osd.44 up 1 >> 45 1.82 osd.45 up 1 >> 46 1.82 osd.46 up 1 >> 47 1.82 osd.47 up 1 >> 48 1.82 osd.48 up 1 >> -9 48.64 room room1 >> -5 15.92 host ceph-3-storage >> 24 1.82 osd.24 up 1 >> 25 1.82 osd.25 up 1 >> 29 1.36 osd.29 up 1 >> 10 3.64 osd.10 up 1 >> 13 3.64 osd.13 up 1 >> 20 3.64 osd.20 up 1 >> -6 20 host ceph-4-storage >> 34 3.64 osd.34 up 1 >> 38 1.36 osd.38 up 1 >> 39 1.36 osd.39 up 1 >> 16 3.64 osd.16 up 1 >> 30 0.68 osd.30 up 1 >> 35 3.64 osd.35 up 1 >> 17 3.64 osd.17 up 1 >> 28 0.68 osd.28 up 1 >> 31 0.68 osd.31 up 1 >> 33 0.68 osd.33 up 1 >> -7 12.72 host ceph-6-storage >> 49 0.45 osd.49 up 1 >> 50 0.45 osd.50 up 1 >> 51 0.45 osd.51 up 1 >> 52 0.45 osd.52 up 1 >> 53 1.82 osd.53 up 1 >> 54 1.82 osd.54 up 1 >> 55 1.82 osd.55 up 1 >> 56 1.82 osd.56 up 1 >> 57 1.82 osd.57 up 1 >> 58 1.82 osd.58 up 1 >> >> What I have tried so far: >> ceph pg repair 2.587 [2.e3 2.c1 2.92] >> ceph pg force_create_pg 2.587 [2.e3 2.c1 2.92] >> ceph osd lost 5 --yes-i-really-mean-it [7 8 13 20] >> >> The history in brief: >> I installed Cuttlefish and updated to Dumpling and to Emperor. The >> Cluster was healthy. Maybe I made ??a mistake during repair of 8 broken >> osds, but from then on I had incompletepgs. At last I have updated from >> Emperor to Firefly. >> >> Regards, >> Mike >> -------------------------------------------------------------------------------------------------- >> Bayerischer Rundfunk; Rundfunkplatz 1; 80335 M?nchen Telefon: +49 89 >> 590001; E-Mail: info at BR.de; Website: http://www.BR.de <http://www.br.de/> >> _______________________________________________ >> ceph-users mailing list >> ceph-users at lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> -------------------------------------------------------------------------------------------------- >> Bayerischer Rundfunk; Rundfunkplatz 1; 80335 M?nchen Telefon: +49 89 >> 590001; E-Mail: info at BR.de; Website: http://www.BR.de <http://www.br.de/> >> >> >> -------------------------------------------------------------------------------------------------- >> Bayerischer Rundfunk; Rundfunkplatz 1; 80335 M?nchen Telefon: +49 89 >> 590001; E-Mail: info at BR.de; Website: http://www.BR.de >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users at lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140813/e8c10006/attachment.htm>
 |