Hi Igor,
> Am 24.04.2020 um 13:09 schrieb Igor Fedotov <ifedotov@xxxxxxx>:
>
> that's not 100% pure experiment. Fresh OSD might be faster by itself. E.g. due to lack of space fragmentation and/or empty lookup tables.
Also the migrated ones were just 3 weeks old having a usage of 5%.
> You might want to recreate OSD.0 without DB and attach DB manually. Then benchmark resulting OSD.
I’ve 24 osds where I migrated db and 8 which where initially created with dB. All of them show the same symptoms.
> Different experiment if you have another slow OSD with recently added DB would be to:
Yes see above 24 of them ;-)
> Compare benchmark results for both bitmap and stupid allocators for this specific OSD. I.e. benchmark it as-is then change bluestore_allocator/bluefs_allocator to stupid and benchmark again.
Can try in a few hours.
> And just in case - I presume all the benchmark results are persistent, i.e. you can see the same results for multiple runs.
Yes I did 10 runs for each posted benchmark.
Thanks,
Stefan
>
>
> Thanks,
>
> Igor
>
>
>
>> On 4/24/2020 12:32 PM, Stefan Priebe - Profihost AG wrote:
>> Hi Igor,
>>
>> there must be a difference. I purged osd.0 and recreated it.
>>
>> Now it gives:
>> ceph tell osd.0 bench
>> {
>> "bytes_written": 1073741824,
>> "blocksize": 4194304,
>> "elapsed_sec": 8.1554735639999993,
>> "bytes_per_sec": 131659040.46819863,
>> "iops": 31.389961354303033
>> }
>>
>> What's wrong wiht adding a block.db device later?
>>
>> Stefan
>>
>>> Am 23.04.20 um 20:34 schrieb Stefan Priebe - Profihost AG:
>>> Hi,
>>>
>>> if the OSDs are idle the difference is even more worse:
>>>
>>> # ceph tell osd.0 bench
>>> {
>>> "bytes_written": 1073741824,
>>> "blocksize": 4194304,
>>> "elapsed_sec": 15.396707875000001,
>>> "bytes_per_sec": 69738403.346825853,
>>> "iops": 16.626931034761871
>>> }
>>>
>>> # ceph tell osd.38 bench
>>> {
>>> "bytes_written": 1073741824,
>>> "blocksize": 4194304,
>>> "elapsed_sec": 6.8903985170000004,
>>> "bytes_per_sec": 155831599.77624846,
>>> "iops": 37.153148597776521
>>> }
>>>
>>> Stefan
>>>
>>> Am 23.04.20 um 14:39 schrieb Stefan Priebe - Profihost AG:
>>>> Hi,
>>>> Am 23.04.20 um 14:06 schrieb Igor Fedotov:
>>>>> I don't recall any additional tuning to be applied to new DB volume. And assume the hardware is pretty the same...
>>>>>
>>>>> Do you still have any significant amount of data spilled over for these updated OSDs? If not I don't have any valid explanation for the phenomena.
>>>>
>>>> just the 64k from here:
>>>> https://tracker.ceph.com/issues/44509
>>>>
>>>>> You might want to try "ceph osd bench" to compare OSDs under pretty the same load. Any difference observed
>>>>
>>>> Servers are the same HW. OSD Bench is:
>>>> # ceph tell osd.0 bench
>>>> {
>>>> "bytes_written": 1073741824,
>>>> "blocksize": 4194304,
>>>> "elapsed_sec": 16.091414781000001,
>>>> "bytes_per_sec": 66727620.822242722,
>>>> "iops": 15.909104543266945
>>>> }
>>>>
>>>> # ceph tell osd.36 bench
>>>> {
>>>> "bytes_written": 1073741824,
>>>> "blocksize": 4194304,
>>>> "elapsed_sec": 10.023828538,
>>>> "bytes_per_sec": 107118933.6419194,
>>>> "iops": 25.539143953780986
>>>> }
>>>>
>>>>
>>>> OSD 0 is a Toshiba MG07SCA12TA SAS 12G
>>>> OSD 36 is a Seagate ST12000NM0008-2H SATA 6G
>>>>
>>>> SSDs are all the same like the rest of the HW. But both drives should give the same performance from their specs. The only other difference is that OSD 36 was directly created with the block.db device (Nautilus 14.2.7) and OSD 0 (14.2.8) does not.
>>>>
>>>> Stefan
>>>>
>>>>>
>>>>> On 4/23/2020 8:35 AM, Stefan Priebe - Profihost AG wrote:
>>>>>> Hello,
>>>>>>
>>>>>> is there anything else needed beside running:
>>>>>> ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-${OSD} bluefs-bdev-new-db --dev-target /dev/vgroup/lvdb-1
>>>>>>
>>>>>> I did so some weeks ago and currently i'm seeing that all osds originally deployed with --block-db show 10-20% I/O waits while all those got converted using ceph-bluestore-tool show 80-100% I/O waits.
>>>>>>
>>>>>> Also is there some tuning available to use more of the SSD? The SSD (block-db) is only saturated at 0-2%.
>>>>>>
>>>>>> Greets,
>>>>>> Stefan
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list -- ceph-users@xxxxxxx
>>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
