---------- Forwarded message ----------
From: Schlacta, Christ <aarcane@xxxxxxxxxxx>
Date: Thu, Feb 23, 2017 at 6:06 PM
Subject: Re: Upgrade Woes on suse leap with OBS ceph.
To: Brad Hubbard <bhubbard@xxxxxxxxxx>
So setting the above to 0 by sheer brute force didn't work, so it's
not crush or osd problem.. also, the errors still say mon0, so I
suspect it's related to communication between libceph in kernel and
the mon.
aarcane@densetsu:/etc/target$ sudo ceph --cluster rk osd crush tunables hammer
adjusted tunables profile to hammer
aarcane@densetsu:/etc/target$ ceph --cluster rk osd crush show-tunables
{
"choose_local_tries": 0,
"choose_local_fallback_tries": 0,
"choose_total_tries": 50,
"chooseleaf_descend_once": 1,
"chooseleaf_vary_r": 1,
"chooseleaf_stable": 0,
"straw_calc_version": 1,
"allowed_bucket_algs": 54,
"profile": "hammer",
"optimal_tunables": 0,
"legacy_tunables": 0,
"minimum_required_version": "firefly",
"require_feature_tunables": 1,
"require_feature_tunables2": 1,
"has_v2_rules": 0,
"require_feature_tunables3": 1,
"has_v3_rules": 0,
"has_v4_buckets": 0,
"require_feature_tunables5": 0,
"has_v5_rules": 0
}
aarcane@densetsu:/etc/target$ sudo rbd --cluster rk map rt1
rbd: sysfs write failed
In some cases useful info is found in syslog - try "dmesg | tail" or so.
rbd: map failed: (110) Connection timed out
aarcane@densetsu:~$ dmesg | tail
[10118.778868] libceph: mon0 10.0.0.67:6789 feature set mismatch, my
40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000
[10118.779597] libceph: mon0 10.0.0.67:6789 missing required protocol features
[10119.834634] libceph: mon0 10.0.0.67:6789 feature set mismatch, my
40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000
[10119.835174] libceph: mon0 10.0.0.67:6789 missing required protocol features
[10120.762983] libceph: mon0 10.0.0.67:6789 feature set mismatch, my
40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000
[10120.763707] libceph: mon0 10.0.0.67:6789 missing required protocol features
[10121.787128] libceph: mon0 10.0.0.67:6789 feature set mismatch, my
40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000
[10121.787847] libceph: mon0 10.0.0.67:6789 missing required protocol features
[10122.911117] libceph: mon0 10.0.0.67:6789 feature set mismatch, my
40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000
[10122.911872] libceph: mon0 10.0.0.67:6789 missing required protocol features
aarcane@densetsu:~$
On Thu, Feb 23, 2017 at 5:56 PM, Schlacta, Christ <aarcane@xxxxxxxxxxx> wrote:
> They're from the suse leap ceph team. They maintain ceph, and build
> up to date versions for suse leap. What I don't know is how to
> disable it. When I try, I get the following mess:
>
> aarcane@densetsu:/etc/target$ ceph --cluster rk osd crush set-tunable
> require_feature_tunables5 0
> Invalid command: require_feature_tunables5 not in straw_calc_version
> osd crush set-tunable straw_calc_version <int> : set crush tunable
> <tunable> to <value>
> Error EINVAL: invalid command
>
> On Thu, Feb 23, 2017 at 5:54 PM, Brad Hubbard <bhubbard@xxxxxxxxxx> wrote:
>> On Fri, Feb 24, 2017 at 11:00 AM, Schlacta, Christ <aarcane@xxxxxxxxxxx> wrote:
>>> aarcane@densetsu:~$ ceph --cluster rk osd crush show-tunables
>>> {
>>> "choose_local_tries": 0,
>>> "choose_local_fallback_tries": 0,
>>> "choose_total_tries": 50,
>>> "chooseleaf_descend_once": 1,
>>> "chooseleaf_vary_r": 1,
>>> "chooseleaf_stable": 1,
>>> "straw_calc_version": 1,
>>> "allowed_bucket_algs": 54,
>>> "profile": "jewel",
>>> "optimal_tunables": 1,
>>> "legacy_tunables": 0,
>>> "minimum_required_version": "jewel",
>>> "require_feature_tunables": 1,
>>> "require_feature_tunables2": 1,
>>> "has_v2_rules": 0,
>>> "require_feature_tunables3": 1,
>>> "has_v3_rules": 0,
>>> "has_v4_buckets": 0,
>>> "require_feature_tunables5": 1,
>>
>> I suspect setting the above to 0 would resolve the issue with the
>> client but there may be a reason why this is set?
>>
>> Where did those packages come from?
>>
>>> "has_v5_rules": 0
>>> }
>>>
>>> On Thu, Feb 23, 2017 at 4:45 PM, Brad Hubbard <bhubbard@xxxxxxxxxx> wrote:
>>>> On Thu, Feb 23, 2017 at 5:18 PM, Schlacta, Christ <aarcane@xxxxxxxxxxx> wrote:
>>>>> So I updated suse leap, and now I'm getting the following error from
>>>>> ceph. I know I need to disable some features, but I'm not sure what
>>>>> they are.. Looks like 14, 57, and 59, but I can't figure out what
>>>>> they correspond to, nor therefore, how to turn them off.
>>>>>
>>>>> libceph: mon0 10.0.0.67:6789 feature set mismatch, my 40106b84a842a42
>>>>> < server's e0106b84a846a42, missing a00000000004000
>>>>
>>>> http://cpp.sh/2rfy says...
>>>>
>>>> Bit 14 set
>>>> Bit 57 set
>>>> Bit 59 set
>>>>
>>>> Comparing this to
>>>> https://github.com/ceph/ceph/blob/master/src/include/ceph_features.h
>>>> shows...
>>>>
>>>> DEFINE_CEPH_FEATURE(14, 2, SERVER_KRAKEN)
>>>> DEFINE_CEPH_FEATURE(57, 1, MON_STATEFUL_SUB)
>>>> DEFINE_CEPH_FEATURE(57, 1, MON_ROUTE_OSDMAP) // overlap
>>>> DEFINE_CEPH_FEATURE(57, 1, OSDSUBOP_NO_SNAPCONTEXT) // overlap
>>>> DEFINE_CEPH_FEATURE(57, 1, SERVER_JEWEL) // overlap
>>>> DEFINE_CEPH_FEATURE(59, 1, FS_BTIME)
>>>> DEFINE_CEPH_FEATURE(59, 1, FS_CHANGE_ATTR) // overlap
>>>> DEFINE_CEPH_FEATURE(59, 1, MSG_ADDR2) // overlap
>>>>
>>>> $ echo "obase=16;ibase=16;$(echo e0106b84a846a42-a00000000004000|tr
>>>> '[a-z]' '[A-Z]')"|bc -qi
>>>> obase=16;ibase=16;E0106B84A846A42-A00000000004000
>>>> 40106B84A842A42
>>>>
>>>> So "me" (the client kernel) does not have the above features that are
>>>> present on the servers.
>>>>
>>>> Can you post the output of "ceph osd crush show-tunables"?
>>>>
>>>>>
>>>>> SuSE Leap 42.2 is Up to date as of tonight, no package updates available.
>>>>> All the ceph packages have the following version:
>>>>>
>>>>> 11.1.0+git.1486588482.ba197ae-72.1
>>>>>
>>>>> And the kernel has version:
>>>>>
>>>>> 4.4.49-16.1
>>>>>
>>>>> It was working perfectly before the upgrade.
>>>>>
>>>>> Thank you very much
>>>>> _______________________________________________
>>>>> ceph-users mailing list
>>>>> ceph-users@xxxxxxxxxxxxxx
>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Brad
>>
>>
>>
>> --
>> Cheers,
>> Brad
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
