On Sun, Mar 17, 2013 at 7:56 PM, Gregory Farnum <greg@xxxxxxxxxxx> wrote: > On Sunday, March 17, 2013 at 4:46 AM, Andrey Korolyov wrote: >> Hi, >> >> from osd tree: >> >> -16 4.95 host 10.5.0.52 >> 32 1.9 osd.32 up 2 >> 33 1.05 osd.33 up 1 >> 34 1 osd.34 up 1 >> 35 1 osd.35 up 1 >> >> df -h: >> /dev/sdd3 3.7T 595G 3.1T 16% /var/lib/ceph/osd/32 >> /dev/sde3 3.7T 332G 3.4T 9% /var/lib/ceph/osd/33 >> /dev/sdf3 3.7T 322G 3.4T 9% /var/lib/ceph/osd/34 >> /dev/sdg3 3.7T 320G 3.4T 9% /var/lib/ceph/osd/35 >> >> -10 2 host 10.5.0.32 >> 18 1 osd.18 up 1 >> 26 1 osd.26 up 1 >> >> df -h: >> /dev/sda2 926G 417G 510G 45% /var/lib/ceph/osd/18 >> /dev/sdb2 926G 431G 496G 47% /var/lib/ceph/osd/26 >> >> Since osds on 10.5.0.32 does not contain garbage bytes almost for >> sure, seems to be some weirdness in the placement. Crush rules are >> almost default, there is no adjustment by node subsets. Any thoughts >> will be appreciated! >> > Do you have any other nodes? What's the rest of your osd tree look like? > > I do note that at a first glance, you've got 1569GB in 10.5.0.52 and 848 in 10.5.0.32, which is a 1.85 differential when you'd really like a ~2.5 differential (based on the very odd CRUSH weights you've assigned to each device, and the hosts). I suspect/hope you've also got something weird going on with the rest of your interior nodes (not pictured here), but perhaps not — and either way I'd recommend fixing up the rest of your weights and seeing if that improves the distribution. Nope, all other osds have weight one(and each host contains two osds, this many-disk system is an experimental one). This host had round values recently, I`ve just changed weights a bit to test a speed of data rearrangement. Problem existed since 10.5.0.52 entered to the data placement with default ``1'' osd weights. > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |