[Hrm, this email was in my spam folder.] At a quick glance, you're probably running into some issues because you've got two racks of very different weights. Things will probably get better if you enable the optimal "crush tunables"; check out the docs on that and see if you can switch to them. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Fri, Dec 27, 2013 at 3:58 AM, Sreejith Vijayendran <sreejith.vijayendran@xxxxxxxxxx> wrote: > Hello, > > We have a 3 node cluster set up with OSDs created on all the 3 nodes. The > replication was set to 2. > 1>> We are in testing phase and tried to bring down all the OSDs in a > particular node and was testing the migration of PGs to other OSDs. > But the PGs are not getting replicated on other OSDs and the status of the > replication also was not clear at all. > Below is the ceph status at that point: > > =============== > sreejith@sb1001:/var/run/ceph$ sudo ceph status > cluster 9b48b60c-bebe-4714-8a61-91ca5b388a17 > health HEALTH_WARN 885 pgs degraded; 885 pgs stuck unclean; recovery > 59/232 objects degraded (25.431%); 22/60 in osds are down > monmap e2: 3 mons at > {sb1001=10.2.4.90:6789/0,sb1002=10.2.4.202:6789/0,sb1004=10.2.4.203:6789/0}, > election epoch 22, quorum 0,1,2 sb1001,sb1002,sb1004 > osdmap e378: 68 osds: 38 up, 60 in > pgmap v4490: 1564 pgs, 25 pools, 1320 MB data, 116 objects > 8415 MB used, 109 TB / 109 TB avail > 59/232 objects degraded (25.431%) > 679 active+clean > 862 active+degraded > 23 active+degraded+remapped > =============== > > We waited for around 4-5 hours and the status just increased marginally to > (25.431%) from (27%) at the start. > > 2>> we then tweaked some OSD values to speed up the recovery > namely(osd_recovery_threads, osd_recovery_max_active, > osd_recovery_max_chunk, osd_max_backfills, osd_backfill_retry_interval etc) > as we were only concerned about getting the OSDs rebalanced as of now. But > this didnt improve at all all over the night till the morning. > > 3>> We then manually started all the OSDs in that specific node and the > status came back up: > But then we could see that there were 23PGs stuck unclean and in > 'active+remapped state' > > ================= > sreejith@sb1001:~$ sudo ceph status > [sudo] password for sreejith: > cluster 9b48b60c-bebe-4714-8a61-91ca5b388a17 > health HEALTH_WARN 23 pgs stuck unclean > monmap e2: 3 mons at > {sb1001=10.2.4.90:6789/0,sb1002=10.2.4.202:6789/0,sb1004=10.2.4.203:6789/0}, > election epoch 22, quorum 0,1,2 sb1001,sb1002,sb1004 > osdmap e382: 68 osds: 61 up, 61 in > pgmap v4931: 1564 pgs, 25 pools, 1320 MB data, 116 objects > 7931 MB used, 110 TB / 110 TB avail > 1541 active+clean > 23 active+remapped > ================= > > The 'pg dump_stuck unclean' was showing that all the PGs were on 4 OSDs and > no other PG were on those same OSDs. > SO: > 4>> we took those OSDs out of the cluster using 'ceph osd out {id}'. Then > the unclean PG number increased to 52. Even after making the OSDs back 'IN', > the situation didn't improve. > > ============= > root@sb1001:/home/sreejith# ceph health detail > HEALTH_WARN 52 pgs stuck unclean > pg 9.63 is stuck unclean since forever, current state active+remapped, last > acting [47,7] > pg 11.61 is stuck unclean since forever, current state active+remapped, last > acting [47,7] > pg 10.62 is stuck unclean since forever, current state active+remapped, last > acting [47,7] > pg 13.5f is stuck unclean since forever, current state active+remapped, last > acting [47,7] > pg 15.5d is stuck unclean since forever, current state active+remapped, last > acting [47,7] > pg 14.5e is stuck unclean since forever, current state active+remapped, last > acting [47,7] > pg 9.47 is stuck unclean for 530.594604, current state active+remapped, last > acting [66,43] > pg 7.49 is stuck unclean for 530.594593, current state active+remapped, last > acting [66,43] > pg 5.4b is stuck unclean for 530.594481, current state active+remapped, last > acting [66,43] > pg 3.4d is stuck unclean for 530.594449, current state active+remapped, last > acting [66,43] > pg 11.45 is stuck unclean for 530.594635, current state active+remapped, > last acting [66,43] > pg 13.43 is stuck unclean for 530.594654, current state active+remapped, > last acting [66,43] > pg 15.41 is stuck unclean for 530.594695, current state active+remapped, > last acting [66,43] > pg 6.4a is stuck unclean for 530.594366, current state active+remapped, last > acting [66,43] > pg 10.46 is stuck unclean for 530.594387, current state active+remapped, > last acting [66,43] > pg 14.42 is stuck unclean for 530.594422, current state active+remapped, > last acting [66,43] > pg 4.4c is stuck unclean for 530.594341, current state active+remapped, last > acting [66,43] > pg 12.44 is stuck unclean for 530.594361, current state active+remapped, > last acting [66,43] > pg 8.48 is stuck unclean for 530.594294, current state active+remapped, last > acting [66,43] > pg 8.30 is stuck unclean for 175428.682512, current state active+remapped, > last acting [51,45] > pg 0.38 is stuck unclean for 175428.682498, current state active+remapped, > last acting [51,45] > pg 7.31 is stuck unclean since forever, current state active+remapped, last > acting [51,45] > pg 5.33 is stuck unclean since forever, current state active+remapped, last > acting [51,45] > pg 3.35 is stuck unclean since forever, current state active+remapped, last > acting [51,45] > pg 1.37 is stuck unclean since forever, current state active+remapped, last > acting [51,45] > pg 6.32 is stuck unclean for 154614.490724, current state active+remapped, > last acting [51,45] > pg 2.36 is stuck unclean since forever, current state active+remapped, last > acting [51,45] > pg 4.34 is stuck unclean since forever, current state active+remapped, last > acting [51,45] > pg 9.24 is stuck unclean since forever, current state active+remapped, last > acting [48,24] > pg 8.25 is stuck unclean for 482.144359, current state active+remapped, last > acting [48,24] > pg 5.28 is stuck unclean since forever, current state active+remapped, last > acting [48,24] > pg 4.29 is stuck unclean since forever, current state active+remapped, last > acting [48,24] > pg 12.21 is stuck unclean since forever, current state active+remapped, last > acting [48,24] > pg 0.2d is stuck unclean for 482.144315, current state active+remapped, last > acting [48,24] > pg 1.2c is stuck unclean since forever, current state active+remapped, last > acting [48,24] > pg 13.20 is stuck unclean since forever, current state active+remapped, last > acting [48,24] > pg 3.2a is stuck unclean since forever, current state active+remapped, last > acting [48,24] > pg 11.22 is stuck unclean since forever, current state active+remapped, last > acting [48,24] > pg 10.23 is stuck unclean since forever, current state active+remapped, last > acting [48,24] > pg 2.2b is stuck unclean since forever, current state active+remapped, last > acting [48,24] > pg 9.2f is stuck unclean since forever, current state active+remapped, last > acting [51,45] > pg 11.2d is stuck unclean since forever, current state active+remapped, last > acting [51,45] > pg 13.2b is stuck unclean since forever, current state active+remapped, last > acting [51,45] > pg 15.29 is stuck unclean since forever, current state active+remapped, last > acting [51,45] > pg 10.2e is stuck unclean since forever, current state active+remapped, last > acting [51,45] > pg 14.2a is stuck unclean since forever, current state active+remapped, last > acting [51,45] > pg 7.26 is stuck unclean since forever, current state active+remapped, last > acting [48,24] > pg 6.27 is stuck unclean since forever, current state active+remapped, last > acting [48,24] > pg 12.2c is stuck unclean since forever, current state active+remapped, last > acting [51,45] > pg 14.1f is stuck unclean since forever, current state active+remapped, last > acting [48,24] > pg 15.1e is stuck unclean since forever, current state active+remapped, last > acting [48,24] > pg 12.60 is stuck unclean for 175428.710580, current state active+remapped, > last acting [47,7] > root@sb1001:/home/sreejith# > ============= > > Now we could see that some other OSDs also came in the list of unclean PGs. > And the number of unclean PGs just increased to 52 > > > So we have two questions: > 1> why isn't the unclean PGs getting cleared, how to debug more. > 2> how to check the recover/backfill status of OSDs.(ceph status, -w, all > commands show same data). It just shows this many are up, in, this many PGs > are clean, remapped. There is no status update of the progress if any. > I referred this bug: > http://tracker.ceph.com/issues/6736 > where the user could see status update like(peering, recovery etc), But for > us these are not shown. > > > More details: > we have all our OSDs of 1.7TB size. > we have 3 nodes in cluster (all 3 having one mon instance each and OSDs on > all) > replication set to default 2 > ceph version 0.72.2 > > Attaching the crush rule set, osd dump, pg dump. > > > -- > > Regards, > Sreejith > > _____________________________________________________________ > The information contained in this communication is intended solely for the > use of the individual or entity to whom it is addressed and others > authorized to receive it. It may contain confidential or legally privileged > information. If you are not the intended recipient you are hereby notified > that any disclosure, copying, distribution or taking any action in reliance > on the contents of this information is strictly prohibited and may be > unlawful. If you have received this communication in error, please notify us > immediately by responding to this email and then delete it from your system. > The firm is neither liable for the proper and complete transmission of the > information contained in this communication nor for any delay in its > receipt. > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |