Hello Guys My CEPH cluster lost data and not its not recovering. This problem occurred when Ceph performed recovery when one of the node was down. Now all the nodes are up but Ceph is showing PG as incomplete , unclean , recovering. I have tried several things to recover them like , scrub , deep-scrub , pg repair , try changing primary affinity and then scrubbing , osd_pool_default_size etc. BUT NO LUCK Could yo please advice , how to recover PG and achieve HEALTH_OK # ceph -s cluster 2bd3283d-67ef-4316-8b7e-d8f4747eae33 health HEALTH_WARN 19 pgs incomplete; 3 pgs recovering; 20 pgs stuck inactive; 23 pgs stuck unclean; 2 requests are blocked > 32 sec; recovery 531/980676 objects degraded (0.054%); 243/326892 unfound (0.074%) monmap e3: 3 mons at {xxx=xxxx:6789/0,xxx=xxxx:6789:6789/0,xxx=xxxx:6789:6789/0}, election epoch 1474, quorum 0,1,2 xx,xx,xx osdmap e261536: 239 osds: 239 up, 238 in pgmap v415790: 18432 pgs, 13 pools, 2330 GB data, 319 kobjects 20316 GB used, 844 TB / 864 TB avail 531/980676 objects degraded (0.054%); 243/326892 unfound (0.074%) 1 creating 18409 active+clean 3 active+recovering 19 incomplete # ceph pg dump_stuck unclean ok pg_stat objects mip degr unf bytes log disklog state state_stamp v reported up up_primary acting acting_primary last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp 10.70 0 0 0 0 0 0 0 incomplete 2015-03-20 12:19:49.534911 0'0 261536:1015 [153,140,80] 153 [153,140,80] 153 0'0 2015-03-12 17:59:43.275049 0'0 2015-03-09 17:55:58.745662 3.dde 68 66 0 66 552861709 297 297 incomplete 2015-03-20 12:19:49.584839 33547'297 261536:228352 [174,5,179] 174 [174,5,179] 174 33547'297 2015-03-12 14:19:15.261595 28522'43 2015-03-11 14:19:13.894538 5.a2 0 0 0 0 0 0 0 incomplete 2015-03-20 12:19:49.560756 0'0 261536:897 [214,191,170] 214 [214,191,170] 214 0'0 2015-03-12 17:58:29.257085 0'0 2015-03-09 17:55:07.684377 13.1b6 0 0 0 0 0 0 0 incomplete 2015-03-20 12:19:49.846253 0'0 261536:1050 [0,176,131] 0 [0,176,131] 0 0'0 2015-03-12 18:00:13.286920 0'0 2015-03-09 17:56:18.715208 7.25b 16 0 0 0 67108864 16 16 incomplete 2015-03-20 12:19:49.639102 27666'16 261536:4777 [194,145,45] 194 [194,145,45] 194 27666'16 2015-03-12 17:59:06.357864 2330'3 2015-03-09 17:55:30.754522 5.19 0 0 0 0 0 0 0 incomplete 2015-03-20 12:19:49.742698 0'0 261536:25410 [212,43,131] 212 [212,43,131] 212 0'0 2015-03-12 13:51:37.777026 0'0 2015-03-11 13:51:35.406246 3.a2f 0 0 0 0 0 0 0 creating 2015-03-20 12:42:15.586372 0'0 0:0 [] -1 [] -1 0'0 0.000000 0'0 0.000000 7.298 0 0 0 0 0 0 0 incomplete 2015-03-20 12:19:49.566966 0'0 261536:900 [187,95,225] 187 [187,95,225] 187 27666'13 2015-03-12 17:59:10.308423 2330'4 2015-03-09 17:55:35.750109 3.a5a 77 87 261 87 623902741 325 325 active+recovering 2015-03-20 10:54:57.443670 33569'325 261536:182464 [150,149,181] 150 [150,149,181] 150 33569'325 2015-03-12 13:58:05.813966 28433'44 2015-03-11 13:57:53.909795 1.1e7 0 0 0 0 0 0 0 incomplete 2015-03-20 12:19:49.610547 0'0 261536:772 [175,182] 175 [175,182] 175 0'0 2015-03-12 17:55:45.203232 0'0 2015-03-09 17:53:49.694822 3.774 79 0 0 0 645136397 339 339 incomplete 2015-03-20 12:19:49.821708 33570'339 261536:166857 [162,39,161] 162 [162,39,161] 162 33570'339 2015-03-12 14:49:03.869447 2226'2 2015-03-09 13:46:49.783950 3.7d0 78 0 0 0 609222686 376 376 incomplete 2015-03-20 12:19:49.534004 33538'376 261536:182810 [117,118,177] 117 [117,118,177] 117 33538'376 2015-03-12 13:51:03.984454 28394'62 2015-03-11 13:50:58.196288 3.d60 0 0 0 0 0 0 0 incomplete 2015-03-20 12:19:49.647196 0'0 261536:833 [154,172,1] 154 [154,172,1] 154 33552'321 2015-03-12 13:44:43.502907 28356'39 2015-03-11 13:44:41.663482 4.1fc 0 0 0 0 0 0 0 incomplete 2015-03-20 12:19:49.610103 0'0 261536:1069 [70,179,58] 70 [70,179,58] 70 0'0 2015-03-12 17:58:19.254170 0'0 2015-03-09 17:54:55.720479 3.e02 72 0 0 0 585105425 304 304 incomplete 2015-03-20 12:19:49.564768 33568'304 261536:167428 [15,102,147] 15 [15,102,147] 15 33568'304 2015-03-16 10:04:19.894789 2246'4 2015-03-09 11:43:44.176331 8.1d4 0 0 0 0 0 0 0 incomplete 2015-03-20 12:19:49.614727 0'0 261536:19611 [126,43,174] 126 [126,43,174] 126 0'0 2015-03-12 14:34:35.258338 0'0 2015-03-12 14:34:35.258338 4.2f4 0 0 0 0 0 0 0 incomplete 2015-03-20 12:19:49.595109 0'0 261536:113791 [181,186,13] 181 [181,186,13] 181 0'0 2015-03-12 14:59:03.529264 0'0 2015-03-09 13:46:40.601301 3.52c 65 23 69 23 543162368 290 290 active+recovering 2015-03-20 10:51:43.664734 33553'290 261536:8431 [212,100,219] 212 [212,100,219] 212 33553'290 2015-03-13 11:44:26.396514 29686'103 2015-03-11 17:18:33.452616 3.e5a 76 70 0 0 623902741 325 325 incomplete 2015-03-20 12:19:49.552071 33569'325 261536:71248 [97,22,62] 97 [97,22,62] 97 33569'325 2015-03-12 13:58:05.813966 28433'44 2015-03-11 13:57:53.909795 8.3a0 0 0 0 0 0 0 0 incomplete 2015-03-20 12:19:49.615728 0'0 261536:173184 [62,14,178] 62 [62,14,178] 62 0'0 2015-03-12 13:52:44.546418 0'0 2015-03-12 13:52:44.546418 3.24e 0 0 0 0 0 0 0 incomplete 2015-03-20 12:19:49.591282 0'0 261536:1026 [103,14,90] 103 [103,14,90] 103 33556'272 2015-03-13 11:44:41.263725 2327'4 2015-03-09 17:54:43.675552 5.f7 0 0 0 0 0 0 0 incomplete 2015-03-20 12:19:49.667823 0'0 261536:853 [73,44,123] 73 [73,44,123] 73 0'0 2015-03-12 17:58:30.257371 0'0 2015-03-09 17:55:11.725629 3.ae8 77 67 201 67 624427024 342 342 active+recovering 2015-03-20 10:50:01.693979 33516'342 261536:149258 [122,144,218] 122 [122,144,218] 122 33516'342 2015-03-12 17:11:01.899062 29638'134 2015-03-11 17:10:59.966372 # PG data is there on multiple OSD’s but Ceph is not recovering the PG , For Example # ceph pg map 7.25b osdmap e261536 pg 7.25b (7.25b) -> up [194,145,45] acting [194,145,45] # ls -l /var/lib/ceph/osd/ceph-194/current/7.25b_head | wc -l 17 # ls -l /var/lib/ceph/osd/ceph-145/current/7.25b_head | wc -l 0 # # ls -l /var/lib/ceph/osd/ceph-45/current/7.25b_head | wc -l 17 Some of the PG are completely lost , i.e they don’t have any data . For example # ceph pg map 10.70 osdmap e261536 pg 10.70 (10.70) -> up [153,140,80] acting [153,140,80] # ls -l /var/lib/ceph/osd/ceph-140/current/10.70_head | wc -l 0 # ls -l /var/lib/ceph/osd/ceph-153/current/10.70_head | wc -l 0 # ls -l /var/lib/ceph/osd/ceph-80/current/10.70_head | wc -l 0 - Karan - |
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
