OK it wasn't the balancer.
It happens after executing all the reweight und crush compat commands:
An even on a much bigger cluster it's 6% again. Some rounding issue? I
migrated with a script so it's not a typo.
Stefan
Am 11.01.2018 um 21:21 schrieb Stefan Priebe - Profihost AG:
> Hi,
>
> Am 11.01.2018 um 21:10 schrieb Sage Weil:
>> On Thu, 11 Jan 2018, Stefan Priebe - Profihost AG wrote:
>>> Am 11.01.2018 um 20:58 schrieb Sage Weil:
>>>> On Thu, 11 Jan 2018, Stefan Priebe - Profihost AG wrote:
>>>>> Hi Sage,
>>>>>
>>>>> this did not work like expected. I tested it in another smaller cluster
>>>>> and it resulted in about 6% misplaced objects.
>>>>
>>>> Can you narrow down at what stage the misplaced objects happened?
>>>
>>> ouch i saw this:
>>> # ceph balancer status
>>> {
>>> "active": true,
>>> "plans": [
>>> "auto_2018-01-11_19:52:28"
>>> ],
>>> "mode": "crush-compat"
>>> }
>>>
>>> so might it be the balancer beeing executed while i was modifying the tree?
>>> Can i stop it and reexecute it manually?
>>
>> You can always 'ceph balancer off'. And I probably wouldn't turn it on
>> until after you've cleaned this up because it will balance with the
>> current weights being the 'target' weights (when in your case they're not
>> (yet)).
>>
>> To manually see what the balancer would do you can
>>
>> ceph balancer optimize foo
>> ceph balancer show foo
>> ceph balancer eval foo # (see numerical analysis)
>>
>> and if it looks good
>>
>> ceph balancer execute foo
>>
>> to actually apply the changes.
>
> ok thanks but it seems there are still leftovers somewhere:
> [expo-office-node1 ~]# ceph balancer optimize stefan
> Error EINVAL: Traceback (most recent call last):
> File "/usr/lib/ceph/mgr/balancer/module.py", line 303, in handle_command
> self.optimize(plan)
> File "/usr/lib/ceph/mgr/balancer/module.py", line 596, in optimize
> return self.do_crush_compat(plan)
> File "/usr/lib/ceph/mgr/balancer/module.py", line 658, in do_crush_compat
> orig_ws = self.get_compat_weight_set_weights()
> File "/usr/lib/ceph/mgr/balancer/module.py", line 837, in
> get_compat_weight_set_weights
> raise RuntimeError('could not find bucket %s' % b['bucket_id'])
> RuntimeError: could not find bucket -6
>
> [expo-office-node1 ~]# ceph osd tree
> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
> -1 14.55957 root default
> -4 3.63989 host expo-office-node1
> 8 ssd 0.90997 osd.8 up 1.00000 1.00000
> 9 ssd 0.90997 osd.9 up 1.00000 1.00000
> 10 ssd 0.90997 osd.10 up 1.00000 1.00000
> 11 ssd 0.90997 osd.11 up 1.00000 1.00000
> -2 3.63989 host expo-office-node2
> 0 ssd 0.90997 osd.0 up 1.00000 1.00000
> 1 ssd 0.90997 osd.1 up 1.00000 1.00000
> 2 ssd 0.90997 osd.2 up 1.00000 1.00000
> 3 ssd 0.90997 osd.3 up 1.00000 1.00000
> -3 3.63989 host expo-office-node3
> 4 ssd 0.90997 osd.4 up 1.00000 1.00000
> 5 ssd 0.90997 osd.5 up 1.00000 1.00000
> 6 ssd 0.90997 osd.6 up 1.00000 1.00000
> 7 ssd 0.90997 osd.7 up 1.00000 1.00000
> -5 3.63989 host expo-office-node4
> 12 ssd 0.90997 osd.12 up 1.00000 1.00000
> 13 ssd 0.90997 osd.13 up 1.00000 1.00000
> 14 ssd 0.90997 osd.14 up 1.00000 1.00000
> 15 ssd 0.90997 osd.15 up 1.00000 1.00000
>
>
> Stefan
>
>> sage
>>
>>
>>>
>>>>
>>>> sage
>>>>
>>>>>
>>>>> Any ideas?
>>>>>
>>>>> Stefan
>>>>> Am 11.01.2018 um 08:09 schrieb Stefan Priebe - Profihost AG:HI
>>>>>> Thanks! Can this be done while still having jewel clients?
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>> Excuse my typo sent from my mobile phone.
>>>>>>
>>>>>> Am 10.01.2018 um 22:56 schrieb Sage Weil <sage@xxxxxxxxxxxx
>>>>>> <mailto:sage@xxxxxxxxxxxx>>:
>>>>>>
>>>>>>> On Wed, 10 Jan 2018, Stefan Priebe - Profihost AG wrote:
>>>>>>>> Am 10.01.2018 um 22:23 schrieb Sage Weil:
>>>>>>>>> On Wed, 10 Jan 2018, Stefan Priebe - Profihost AG wrote:
>>>>>>>>>> k,
>>>>>>>>>>
>>>>>>>>>> in the past we used the python crush optimize tool to reweight the osd
>>>>>>>>>> usage - it inserted a 2nd tree with $hostname-target-weight as
>>>>>>>>>> hostnames.
>>>>>>>>>
>>>>>>>>> Can you attach a 'ceph osd crush tree' (or partial output) so I can see
>>>>>>>>> what you mean?
>>>>>>>>
>>>>>>>> Sure - attached.
>>>>>>>
>>>>>>> Got it
>>>>>>>
>>>>>>>>>> Now the quesions are:
>>>>>>>>>> 1.) can we remove the tree? How?
>>>>>>>>>> 2.) Can we do this now or only after all clients are running Luminous?
>>>>>>>>>> 3.) is it enought to enable the mgr balancer module?
>>>>>>>
>>>>>>> First,
>>>>>>>
>>>>>>> ceph osd crush weight-set create-compat
>>>>>>>
>>>>>>> then for each osd,
>>>>>>> ceph osd crush weight-set reweight-compat <osd> <optimized-weight>
>>>>>>> ceph osd crush reweight <osd> <target-weight>
>>>>>>>
>>>>>>> That won't move any data but will keep your current optimized weights in
>>>>>>> the compat weight-set where they belong.
>>>>>>>
>>>>>>> Then you can remove the *-target-weight buckets. For each osd,
>>>>>>>
>>>>>>> ceph osd crush rm <osd> <ancestor>-target-weight
>>>>>>>
>>>>>>> and then for each remaining bucket
>>>>>>>
>>>>>>> ceph osd crush rm <foo>-target-weight
>>>>>>>
>>>>>>> Finally, turn on the balancer (or test it to see what it it wants to do
>>>>>>> with the optimize command.)
>>>>>>>
>>>>>>> HTH!
>>>>>>> sage
>>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
