On Fri, Aug 29, 2014 at 2:53 AM, Christian Balzer <chibi at gol.com> wrote: >> Now, 1200 is not a power of two, but it makes sense. (12 x 100). > Should have been 600 and then upped to 1024. At the time, there was a reason why doing that did not work, but I don't remember the specifics. All messages sent back in time telling then-us to try harder or make better notes have thusfar been ignored. >> Probably we forewent the power of two because it was such a huge >> increase and we were already erring large. >> > Which unfortunately in my experience is what you have to do if you want > even distribution with smallish clusters. In the end, this made no difference. By slipping one more OSD into the fray, I was able to bring the average utilization down enough to inch up to 2048 PG's. It had basically no effect on how evenly the OSD's are used. (Counting the new OSD, which is only 62% used, things have actually gotten worse.) Here are the current df's: Node 1: /dev/sda2 358G 269G 89G 76% /var/lib/ceph/osd/ceph-0 /dev/sdb2 358G 310G 49G 87% /var/lib/ceph/osd/ceph-1 /dev/sdc2 358G 286G 73G 80% /var/lib/ceph/osd/ceph-2 /dev/sdd2 358G 287G 71G 81% /var/lib/ceph/osd/ceph-3 Node 2: /dev/sda2 358G 288G 70G 81% /var/lib/ceph/osd/ceph-4 /dev/sdd2 358G 311G 48G 87% /var/lib/ceph/osd/ceph-9 /dev/sdc2 358G 278G 81G 78% /var/lib/ceph/osd/ceph-10 /dev/sdb2 358G 296G 62G 83% /var/lib/ceph/osd/ceph-11 Node 3: /dev/sda2 358G 291G 67G 82% /var/lib/ceph/osd/ceph-5 /dev/sdb2 358G 296G 63G 83% /var/lib/ceph/osd/ceph-6 /dev/sdc2 358G 298G 61G 84% /var/lib/ceph/osd/ceph-7 /dev/sdd2 358G 282G 77G 79% /var/lib/ceph/osd/ceph-8 Node 4: /dev/sdb2 358G 219G 140G 62% /var/lib/ceph/osd/ceph-12 > If you're using RBD to for VM images, you might be able to get space back > by doing a fstrim on those images from inside the VM. This isn't really about getting space back; we can buy more space if we need it. It's about not having stuff (like backfilling) fail because a 1-2 OSDs are at 87% when the average use is <80%. So it seems like we're back to square one in terms of balancing out our OSD's. Is there a way to do it? Thanks!
 |