On Fri, Feb 24, 2017 at 3:07 PM, Schlacta, Christ <aarcane@xxxxxxxxxxx> wrote: > So hopefully when the suse ceph team get 11.2 released it should fix this, > yes? Definitely not a question I can answer. What I can tell you is the fix is only in master atm, not yet backported to kraken http://tracker.ceph.com/issues/18842 > > On Feb 23, 2017 21:06, "Brad Hubbard" <bhubbard@xxxxxxxxxx> wrote: >> >> Kefu has just pointed out that this has the hallmarks of >> https://github.com/ceph/ceph/pull/13275 >> >> On Fri, Feb 24, 2017 at 3:00 PM, Brad Hubbard <bhubbard@xxxxxxxxxx> wrote: >> > Hmm, >> > >> > What's interesting is the feature set reported by the servers has only >> > changed from >> > >> > e0106b84a846a42 >> > >> > Bit 1 set Bit 6 set Bit 9 set Bit 11 set Bit 13 set Bit 14 set Bit 18 >> > set Bit 23 set Bit 25 set Bit 27 set Bit 30 set Bit 35 set Bit 36 set >> > Bit 37 set Bit 39 set Bit 41 set Bit 42 set Bit 48 set Bit 57 set Bit >> > 58 set Bit 59 set >> > >> > to >> > >> > e0106b84a846a52 >> > >> > Bit 1 set Bit 4 set Bit 6 set Bit 9 set Bit 11 set Bit 13 set Bit 14 >> > set Bit 18 set Bit 23 set Bit 25 set Bit 27 set Bit 30 set Bit 35 set >> > Bit 36 set Bit 37 set Bit 39 set Bit 41 set Bit 42 set Bit 48 set Bit >> > 57 set Bit 58 set Bit 59 set >> > >> > So all it's done is *added* Bit 4 which is DEFINE_CEPH_FEATURE( 4, 1, >> > SUBSCRIBE2) >> > >> > >> > On Fri, Feb 24, 2017 at 1:40 PM, Schlacta, Christ <aarcane@xxxxxxxxxxx> >> > wrote: >> >> # begin crush map >> >> tunable choose_local_tries 0 >> >> tunable choose_local_fallback_tries 0 >> >> tunable choose_total_tries 50 >> >> tunable chooseleaf_descend_once 1 >> >> tunable chooseleaf_vary_r 1 >> >> tunable straw_calc_version 1 >> >> tunable allowed_bucket_algs 54 >> >> >> >> # devices >> >> device 0 osd.0 >> >> device 1 osd.1 >> >> device 2 osd.2 >> >> >> >> # types >> >> type 0 osd >> >> type 1 host >> >> type 2 chassis >> >> type 3 rack >> >> type 4 row >> >> type 5 pdu >> >> type 6 pod >> >> type 7 room >> >> type 8 datacenter >> >> type 9 region >> >> type 10 root >> >> >> >> # buckets >> >> host densetsu { >> >> id -2 # do not change unnecessarily >> >> # weight 0.293 >> >> alg straw >> >> hash 0 # rjenkins1 >> >> item osd.0 weight 0.146 >> >> item osd.1 weight 0.146 >> >> } >> >> host density { >> >> id -3 # do not change unnecessarily >> >> # weight 0.145 >> >> alg straw >> >> hash 0 # rjenkins1 >> >> item osd.2 weight 0.145 >> >> } >> >> root default { >> >> id -1 # do not change unnecessarily >> >> # weight 0.438 >> >> alg straw >> >> hash 0 # rjenkins1 >> >> item densetsu weight 0.293 >> >> item density weight 0.145 >> >> } >> >> >> >> # rules >> >> rule replicated_ruleset { >> >> ruleset 0 >> >> type replicated >> >> min_size 1 >> >> max_size 10 >> >> step take default >> >> step chooseleaf firstn 0 type host >> >> step emit >> >> } >> >> >> >> # end crush map >> >> >> >> On Thu, Feb 23, 2017 at 7:37 PM, Brad Hubbard <bhubbard@xxxxxxxxxx> >> >> wrote: >> >>> Did you dump out the crushmap and look? >> >>> >> >>> On Fri, Feb 24, 2017 at 1:36 PM, Schlacta, Christ >> >>> <aarcane@xxxxxxxxxxx> wrote: >> >>>> insofar as I can tell, yes. Everything indicates that they are in >> >>>> effect. >> >>>> >> >>>> On Thu, Feb 23, 2017 at 7:14 PM, Brad Hubbard <bhubbard@xxxxxxxxxx> >> >>>> wrote: >> >>>>> Is your change reflected in the current crushmap? >> >>>>> >> >>>>> On Fri, Feb 24, 2017 at 12:07 PM, Schlacta, Christ >> >>>>> <aarcane@xxxxxxxxxxx> wrote: >> >>>>>> ---------- Forwarded message ---------- >> >>>>>> From: Schlacta, Christ <aarcane@xxxxxxxxxxx> >> >>>>>> Date: Thu, Feb 23, 2017 at 6:06 PM >> >>>>>> Subject: Re: Upgrade Woes on suse leap with OBS ceph. >> >>>>>> To: Brad Hubbard <bhubbard@xxxxxxxxxx> >> >>>>>> >> >>>>>> >> >>>>>> So setting the above to 0 by sheer brute force didn't work, so it's >> >>>>>> not crush or osd problem.. also, the errors still say mon0, so I >> >>>>>> suspect it's related to communication between libceph in kernel and >> >>>>>> the mon. >> >>>>>> >> >>>>>> aarcane@densetsu:/etc/target$ sudo ceph --cluster rk osd crush >> >>>>>> tunables hammer >> >>>>>> adjusted tunables profile to hammer >> >>>>>> aarcane@densetsu:/etc/target$ ceph --cluster rk osd crush >> >>>>>> show-tunables >> >>>>>> { >> >>>>>> "choose_local_tries": 0, >> >>>>>> "choose_local_fallback_tries": 0, >> >>>>>> "choose_total_tries": 50, >> >>>>>> "chooseleaf_descend_once": 1, >> >>>>>> "chooseleaf_vary_r": 1, >> >>>>>> "chooseleaf_stable": 0, >> >>>>>> "straw_calc_version": 1, >> >>>>>> "allowed_bucket_algs": 54, >> >>>>>> "profile": "hammer", >> >>>>>> "optimal_tunables": 0, >> >>>>>> "legacy_tunables": 0, >> >>>>>> "minimum_required_version": "firefly", >> >>>>>> "require_feature_tunables": 1, >> >>>>>> "require_feature_tunables2": 1, >> >>>>>> "has_v2_rules": 0, >> >>>>>> "require_feature_tunables3": 1, >> >>>>>> "has_v3_rules": 0, >> >>>>>> "has_v4_buckets": 0, >> >>>>>> "require_feature_tunables5": 0, >> >>>>>> "has_v5_rules": 0 >> >>>>>> } >> >>>>>> >> >>>>>> aarcane@densetsu:/etc/target$ sudo rbd --cluster rk map rt1 >> >>>>>> rbd: sysfs write failed >> >>>>>> In some cases useful info is found in syslog - try "dmesg | tail" >> >>>>>> or so. >> >>>>>> rbd: map failed: (110) Connection timed out >> >>>>>> aarcane@densetsu:~$ dmesg | tail >> >>>>>> [10118.778868] libceph: mon0 10.0.0.67:6789 feature set mismatch, >> >>>>>> my >> >>>>>> 40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000 >> >>>>>> [10118.779597] libceph: mon0 10.0.0.67:6789 missing required >> >>>>>> protocol features >> >>>>>> [10119.834634] libceph: mon0 10.0.0.67:6789 feature set mismatch, >> >>>>>> my >> >>>>>> 40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000 >> >>>>>> [10119.835174] libceph: mon0 10.0.0.67:6789 missing required >> >>>>>> protocol features >> >>>>>> [10120.762983] libceph: mon0 10.0.0.67:6789 feature set mismatch, >> >>>>>> my >> >>>>>> 40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000 >> >>>>>> [10120.763707] libceph: mon0 10.0.0.67:6789 missing required >> >>>>>> protocol features >> >>>>>> [10121.787128] libceph: mon0 10.0.0.67:6789 feature set mismatch, >> >>>>>> my >> >>>>>> 40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000 >> >>>>>> [10121.787847] libceph: mon0 10.0.0.67:6789 missing required >> >>>>>> protocol features >> >>>>>> [10122.911117] libceph: mon0 10.0.0.67:6789 feature set mismatch, >> >>>>>> my >> >>>>>> 40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000 >> >>>>>> [10122.911872] libceph: mon0 10.0.0.67:6789 missing required >> >>>>>> protocol features >> >>>>>> aarcane@densetsu:~$ >> >>>>>> >> >>>>>> >> >>>>>> On Thu, Feb 23, 2017 at 5:56 PM, Schlacta, Christ >> >>>>>> <aarcane@xxxxxxxxxxx> wrote: >> >>>>>>> They're from the suse leap ceph team. They maintain ceph, and >> >>>>>>> build >> >>>>>>> up to date versions for suse leap. What I don't know is how to >> >>>>>>> disable it. When I try, I get the following mess: >> >>>>>>> >> >>>>>>> aarcane@densetsu:/etc/target$ ceph --cluster rk osd crush >> >>>>>>> set-tunable >> >>>>>>> require_feature_tunables5 0 >> >>>>>>> Invalid command: require_feature_tunables5 not in >> >>>>>>> straw_calc_version >> >>>>>>> osd crush set-tunable straw_calc_version <int> : set crush >> >>>>>>> tunable >> >>>>>>> <tunable> to <value> >> >>>>>>> Error EINVAL: invalid command >> >>>>>>> >> >>>>>>> On Thu, Feb 23, 2017 at 5:54 PM, Brad Hubbard >> >>>>>>> <bhubbard@xxxxxxxxxx> wrote: >> >>>>>>>> On Fri, Feb 24, 2017 at 11:00 AM, Schlacta, Christ >> >>>>>>>> <aarcane@xxxxxxxxxxx> wrote: >> >>>>>>>>> aarcane@densetsu:~$ ceph --cluster rk osd crush show-tunables >> >>>>>>>>> { >> >>>>>>>>> "choose_local_tries": 0, >> >>>>>>>>> "choose_local_fallback_tries": 0, >> >>>>>>>>> "choose_total_tries": 50, >> >>>>>>>>> "chooseleaf_descend_once": 1, >> >>>>>>>>> "chooseleaf_vary_r": 1, >> >>>>>>>>> "chooseleaf_stable": 1, >> >>>>>>>>> "straw_calc_version": 1, >> >>>>>>>>> "allowed_bucket_algs": 54, >> >>>>>>>>> "profile": "jewel", >> >>>>>>>>> "optimal_tunables": 1, >> >>>>>>>>> "legacy_tunables": 0, >> >>>>>>>>> "minimum_required_version": "jewel", >> >>>>>>>>> "require_feature_tunables": 1, >> >>>>>>>>> "require_feature_tunables2": 1, >> >>>>>>>>> "has_v2_rules": 0, >> >>>>>>>>> "require_feature_tunables3": 1, >> >>>>>>>>> "has_v3_rules": 0, >> >>>>>>>>> "has_v4_buckets": 0, >> >>>>>>>>> "require_feature_tunables5": 1, >> >>>>>>>> >> >>>>>>>> I suspect setting the above to 0 would resolve the issue with the >> >>>>>>>> client but there may be a reason why this is set? >> >>>>>>>> >> >>>>>>>> Where did those packages come from? >> >>>>>>>> >> >>>>>>>>> "has_v5_rules": 0 >> >>>>>>>>> } >> >>>>>>>>> >> >>>>>>>>> On Thu, Feb 23, 2017 at 4:45 PM, Brad Hubbard >> >>>>>>>>> <bhubbard@xxxxxxxxxx> wrote: >> >>>>>>>>>> On Thu, Feb 23, 2017 at 5:18 PM, Schlacta, Christ >> >>>>>>>>>> <aarcane@xxxxxxxxxxx> wrote: >> >>>>>>>>>>> So I updated suse leap, and now I'm getting the following >> >>>>>>>>>>> error from >> >>>>>>>>>>> ceph. I know I need to disable some features, but I'm not >> >>>>>>>>>>> sure what >> >>>>>>>>>>> they are.. Looks like 14, 57, and 59, but I can't figure out >> >>>>>>>>>>> what >> >>>>>>>>>>> they correspond to, nor therefore, how to turn them off. >> >>>>>>>>>>> >> >>>>>>>>>>> libceph: mon0 10.0.0.67:6789 feature set mismatch, my >> >>>>>>>>>>> 40106b84a842a42 >> >>>>>>>>>>> < server's e0106b84a846a42, missing a00000000004000 >> >>>>>>>>>> >> >>>>>>>>>> http://cpp.sh/2rfy says... >> >>>>>>>>>> >> >>>>>>>>>> Bit 14 set >> >>>>>>>>>> Bit 57 set >> >>>>>>>>>> Bit 59 set >> >>>>>>>>>> >> >>>>>>>>>> Comparing this to >> >>>>>>>>>> >> >>>>>>>>>> https://github.com/ceph/ceph/blob/master/src/include/ceph_features.h >> >>>>>>>>>> shows... >> >>>>>>>>>> >> >>>>>>>>>> DEFINE_CEPH_FEATURE(14, 2, SERVER_KRAKEN) >> >>>>>>>>>> DEFINE_CEPH_FEATURE(57, 1, MON_STATEFUL_SUB) >> >>>>>>>>>> DEFINE_CEPH_FEATURE(57, 1, MON_ROUTE_OSDMAP) // overlap >> >>>>>>>>>> DEFINE_CEPH_FEATURE(57, 1, OSDSUBOP_NO_SNAPCONTEXT) // overlap >> >>>>>>>>>> DEFINE_CEPH_FEATURE(57, 1, SERVER_JEWEL) // overlap >> >>>>>>>>>> DEFINE_CEPH_FEATURE(59, 1, FS_BTIME) >> >>>>>>>>>> DEFINE_CEPH_FEATURE(59, 1, FS_CHANGE_ATTR) // overlap >> >>>>>>>>>> DEFINE_CEPH_FEATURE(59, 1, MSG_ADDR2) // overlap >> >>>>>>>>>> >> >>>>>>>>>> $ echo "obase=16;ibase=16;$(echo >> >>>>>>>>>> e0106b84a846a42-a00000000004000|tr >> >>>>>>>>>> '[a-z]' '[A-Z]')"|bc -qi >> >>>>>>>>>> obase=16;ibase=16;E0106B84A846A42-A00000000004000 >> >>>>>>>>>> 40106B84A842A42 >> >>>>>>>>>> >> >>>>>>>>>> So "me" (the client kernel) does not have the above features >> >>>>>>>>>> that are >> >>>>>>>>>> present on the servers. >> >>>>>>>>>> >> >>>>>>>>>> Can you post the output of "ceph osd crush show-tunables"? >> >>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> SuSE Leap 42.2 is Up to date as of tonight, no package updates >> >>>>>>>>>>> available. >> >>>>>>>>>>> All the ceph packages have the following version: >> >>>>>>>>>>> >> >>>>>>>>>>> 11.1.0+git.1486588482.ba197ae-72.1 >> >>>>>>>>>>> >> >>>>>>>>>>> And the kernel has version: >> >>>>>>>>>>> >> >>>>>>>>>>> 4.4.49-16.1 >> >>>>>>>>>>> >> >>>>>>>>>>> It was working perfectly before the upgrade. >> >>>>>>>>>>> >> >>>>>>>>>>> Thank you very much >> >>>>>>>>>>> _______________________________________________ >> >>>>>>>>>>> ceph-users mailing list >> >>>>>>>>>>> ceph-users@xxxxxxxxxxxxxx >> >>>>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> -- >> >>>>>>>>>> Cheers, >> >>>>>>>>>> Brad >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>>> Cheers, >> >>>>>>>> Brad >> >>>>>> _______________________________________________ >> >>>>>> ceph-users mailing list >> >>>>>> ceph-users@xxxxxxxxxxxxxx >> >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> Cheers, >> >>>>> Brad >> >>> >> >>> >> >>> >> >>> -- >> >>> Cheers, >> >>> Brad >> > >> > >> > >> > -- >> > Cheers, >> > Brad >> >> >> >> -- >> Cheers, >> Brad -- Cheers, Brad _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |