Hi Greg,
Following are the test outcomes on EC profile ( n = k + m)
1. Kraken filestore and bluetore with m=1 , recovery does not start .
2. Jewel filestore and bluestore with m=1 , recovery happens .
3. Kraken bluestore all default configuration and m=1, no recovery.
4. Kraken bluestore with m=2 , recovery happens when one OSD is down and for 2 OSD fails.
So, the issue seems to be on ceph-kraken release. Your views…
Thanks,
Muthu
On 31 January 2017 at 14:18, Muthusamy Muthiah <muthiah.muthusamy@xxxxxxxxx> wrote:
Hi Greg,Now we could see the same problem exists for kraken-filestore also.Attached the requested osdmap and crushmap.OSD.1 was stopped in this following procedure and OSD map for a PG is displayed.ceph osd dump | grep cdvr_ec2017-01-31 08:39:44.827079 7f323d66c700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb2017-01-31 08:39:44.848901 7f323d66c700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdbpool 2 'cdvr_ec' erasure size 4 min_size 4 crush_ruleset 1 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 234 flags hashpspool stripe_width 4128[root@ca-cn2 ~]# ceph osd getmap -o /tmp/osdmap[root@ca-cn2 ~]# osdmaptool --pool 2 --test-map-object object1 /tmp/osdmaposdmaptool: osdmap file '/tmp/osdmap'object 'object1' -> 2.2bc -> [20,47,1,36][root@ca-cn2 ~]# ceph osd map cdvr_ec object1osdmap e402 pool 'cdvr_ec' (2) object 'object1' -> pg 2.bac5debc (2.2bc) -> up ([20,47,1,36], p20) acting ([20,47,1,36], p20)[root@ca-cn2 ~]# systemctl stop ceph-osd@1.service[root@ca-cn2 ~]# ceph osd getmap -o /tmp/osdmap1[root@ca-cn2 ~]# osdmaptool --pool 2 --test-map-object object1 /tmp/osdmap1osdmaptool: osdmap file '/tmp/osdmap1'object 'object1' -> 2.2bc -> [20,47,2147483647,36][root@ca-cn2 ~]# ceph osd map cdvr_ec object1osdmap e406 pool 'cdvr_ec' (2) object 'object1' -> pg 2.bac5debc (2.2bc) -> up ([20,47,39,36], p20) acting ([20,47,NONE,36], p20)[root@ca-cn2 ~]# ceph osd tree2017-01-31 08:42:19.606876 7f4ed856a700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb2017-01-31 08:42:19.628358 7f4ed856a700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdbID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY-1 327.47314 root default-2 65.49463 host ca-cn43 5.45789 osd.3 up 1.00000 1.000005 5.45789 osd.5 up 1.00000 1.0000010 5.45789 osd.10 up 1.00000 1.0000016 5.45789 osd.16 up 1.00000 1.0000021 5.45789 osd.21 up 1.00000 1.0000027 5.45789 osd.27 up 1.00000 1.0000030 5.45789 osd.30 up 1.00000 1.0000035 5.45789 osd.35 up 1.00000 1.0000042 5.45789 osd.42 up 1.00000 1.0000047 5.45789 osd.47 up 1.00000 1.0000051 5.45789 osd.51 up 1.00000 1.0000053 5.45789 osd.53 up 1.00000 1.00000-3 65.49463 host ca-cn32 5.45789 osd.2 up 1.00000 1.000006 5.45789 osd.6 up 1.00000 1.0000011 5.45789 osd.11 up 1.00000 1.0000015 5.45789 osd.15 up 1.00000 1.0000020 5.45789 osd.20 up 1.00000 1.0000025 5.45789 osd.25 up 1.00000 1.0000029 5.45789 osd.29 up 1.00000 1.0000033 5.45789 osd.33 up 1.00000 1.0000038 5.45789 osd.38 up 1.00000 1.0000040 5.45789 osd.40 up 1.00000 1.0000045 5.45789 osd.45 up 1.00000 1.0000049 5.45789 osd.49 up 1.00000 1.00000-4 65.49463 host ca-cn50 5.45789 osd.0 up 1.00000 1.000007 5.45789 osd.7 up 1.00000 1.0000012 5.45789 osd.12 up 1.00000 1.0000017 5.45789 osd.17 up 1.00000 1.0000023 5.45789 osd.23 up 1.00000 1.0000026 5.45789 osd.26 up 1.00000 1.0000032 5.45789 osd.32 up 1.00000 1.0000034 5.45789 osd.34 up 1.00000 1.0000041 5.45789 osd.41 up 1.00000 1.0000046 5.45789 osd.46 up 1.00000 1.0000052 5.45789 osd.52 up 1.00000 1.0000056 5.45789 osd.56 up 1.00000 1.00000-5 65.49463 host ca-cn14 5.45789 osd.4 up 1.00000 1.000009 5.45789 osd.9 up 1.00000 1.0000014 5.45789 osd.14 up 1.00000 1.0000019 5.45789 osd.19 up 1.00000 1.0000024 5.45789 osd.24 up 1.00000 1.0000036 5.45789 osd.36 up 1.00000 1.0000043 5.45789 osd.43 up 1.00000 1.0000050 5.45789 osd.50 up 1.00000 1.0000055 5.45789 osd.55 up 1.00000 1.0000057 5.45789 osd.57 up 1.00000 1.0000058 5.45789 osd.58 up 1.00000 1.0000059 5.45789 osd.59 up 1.00000 1.00000-6 65.49463 host ca-cn21 5.45789 osd.1 down 0 1.000008 5.45789 osd.8 up 1.00000 1.0000013 5.45789 osd.13 up 1.00000 1.0000018 5.45789 osd.18 up 1.00000 1.0000022 5.45789 osd.22 up 1.00000 1.0000028 5.45789 osd.28 up 1.00000 1.0000031 5.45789 osd.31 up 1.00000 1.0000037 5.45789 osd.37 up 1.00000 1.0000039 5.45789 osd.39 up 1.00000 1.0000044 5.45789 osd.44 up 1.00000 1.0000048 5.45789 osd.48 up 1.00000 1.0000054 5.45789 osd.54 up 1.00000 1.00000health HEALTH_ERR69 pgs are stuck inactive for more than 300 seconds69 pgs incomplete69 pgs stuck inactive69 pgs stuck unclean512 requests are blocked > 32 secelection epoch 8, quorum 0,1,2,3,4 ca-cn1,ca-cn2,ca-cn3,ca-cn4,ca-cn5 mgr active: ca-cn4 standbys: ca-cn2, ca-cn5, ca-cn3, ca-cn1osdmap e406: 60 osds: 59 up, 59 in; 69 remapped pgsflags sortbitwise,require_jewel_osds,require_kraken_osds pgmap v23018: 1024 pgs, 1 pools, 3892 GB data, 7910 kobjects6074 GB used, 316 TB / 322 TB avail955 active+clean69 remapped+incompleteThanks,MuthuOn 31 January 2017 at 02:54, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:You might also check out "ceph osd tree" and crush dump and make sure
they look the way you expect.
On Mon, Jan 30, 2017 at 1:23 PM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
> On Sun, Jan 29, 2017 at 6:40 AM, Muthusamy Muthiah
> <muthiah.muthusamy@xxxxxxxxx> wrote:
>> Hi All,
>>
>> Also tried EC profile 3+1 on 5 node cluster with bluestore enabled . When
>> an OSD is down the cluster goes to ERROR state even when the cluster is n+1
>> . No recovery happening.
>>
>> health HEALTH_ERR
>> 75 pgs are stuck inactive for more than 300 seconds
>> 75 pgs incomplete
>> 75 pgs stuck inactive
>> 75 pgs stuck unclean
>> monmap e2: 5 mons at
>> {ca-cn1=10.50.5.117:6789/0,ca-cn2=10.50.5.118:6789/0,ca-cn3= }10.50.5.119:6789/0,ca-cn4=10.5 0.5.120:6789/0,ca-cn5=10.50.5. 121:6789/0
>> election epoch 10, quorum 0,1,2,3,4
>> ca-cn1,ca-cn2,ca-cn3,ca-cn4,ca-cn5
>> mgr active: ca-cn1 standbys: ca-cn4, ca-cn3, ca-cn5, ca-cn2
>> osdmap e264: 60 osds: 59 up, 59 in; 75 remapped pgs
>> flags sortbitwise,require_jewel_osds,require_kraken_osds
>> pgmap v119402: 1024 pgs, 1 pools, 28519 GB data, 21548 kobjects
>> 39976 GB used, 282 TB / 322 TB avail
>> 941 active+clean
>> 75 remapped+incomplete
>> 8 active+clean+scrubbing
>>
>> this seems to be an issue with bluestore , recovery not happening properly
>> with EC .
>
> It's possible but it seems a lot more likely this is some kind of
> config issue. Can you share your osd map ("ceph osd getmap")?
> -Greg
>
>>
>> Thanks,
>> Muthu
>>
>> On 24 January 2017 at 12:57, Muthusamy Muthiah <muthiah.muthusamy@xxxxxxxxx>
>> wrote:
>>>
>>> Hi Greg,
>>>
>>> We use EC:4+1 on 5 node cluster in production deployments with filestore
>>> and it does recovery and peering when one OSD goes down. After few mins ,
>>> other OSD from a node where the fault OSD exists will take over the PGs
>>> temporarily and all PGs goes to active + clean state . Cluster also does not
>>> goes down during this recovery process.
>>>
>>> Only on bluestore we see cluster going to error state when one OSD is
>>> down.
>>> We are still validating this and let you know additional findings.
>>>
>>> Thanks,
>>> Muthu
>>>
>>> On 21 January 2017 at 02:06, Shinobu Kinjo <skinjo@xxxxxxxxxx> wrote:
>>>>
>>>> `ceph pg dump` should show you something like:
>>>>
>>>> * active+undersized+degraded ... [NONE,3,2,4,1] 3 [NONE,3,2,4,1]
>>>>
>>>> Sam,
>>>>
>>>> Am I wrong? Or is it up to something else?
>>>>
>>>>
>>>> On Sat, Jan 21, 2017 at 4:22 AM, Gregory Farnum <gfarnum@xxxxxxxxxx>
>>>> wrote:
>>>> > I'm pretty sure the default configs won't let an EC PG go active with
>>>> > only "k" OSDs in its PG; it needs at least k+1 (or possibly more? Not
>>>> > certain). Running an "n+1" EC config is just not a good idea.
>>>> > For testing you could probably adjust this with the equivalent of
>>>> > min_size for EC pools, but I don't know the parameters off the top of
>>>> > my head.
>>>> > -Greg
>>>> >
>>>> > On Fri, Jan 20, 2017 at 2:15 AM, Muthusamy Muthiah
>>>> > <muthiah.muthusamy@xxxxxxxxx> wrote:
>>>> >> Hi ,
>>>> >>
>>>> >> We are validating kraken 11.2.0 with bluestore on 5 node cluster with
>>>> >> EC
>>>> >> 4+1.
>>>> >>
>>>> >> When an OSD is down , the peering is not happening and ceph health
>>>> >> status
>>>> >> moved to ERR state after few mins. This was working in previous
>>>> >> development
>>>> >> releases. Any additional configuration required in v11.2.0
>>>> >>
>>>> >> Following is our ceph configuration:
>>>> >>
>>>> >> mon_osd_down_out_interval = 30
>>>> >> mon_osd_report_timeout = 30
>>>> >> mon_osd_down_out_subtree_limit = host
>>>> >> mon_osd_reporter_subtree_level = host
>>>> >>
>>>> >> and the recovery parameters set to default.
>>>> >>
>>>> >> [root@ca-cn1 ceph]# ceph osd crush show-tunables
>>>> >>
>>>> >> {
>>>> >> "choose_local_tries": 0,
>>>> >> "choose_local_fallback_tries": 0,
>>>> >> "choose_total_tries": 50,
>>>> >> "chooseleaf_descend_once": 1,
>>>> >> "chooseleaf_vary_r": 1,
>>>> >> "chooseleaf_stable": 1,
>>>> >> "straw_calc_version": 1,
>>>> >> "allowed_bucket_algs": 54,
>>>> >> "profile": "jewel",
>>>> >> "optimal_tunables": 1,
>>>> >> "legacy_tunables": 0,
>>>> >> "minimum_required_version": "jewel",
>>>> >> "require_feature_tunables": 1,
>>>> >> "require_feature_tunables2": 1,
>>>> >> "has_v2_rules": 1,
>>>> >> "require_feature_tunables3": 1,
>>>> >> "has_v3_rules": 0,
>>>> >> "has_v4_buckets": 0,
>>>> >> "require_feature_tunables5": 1,
>>>> >> "has_v5_rules": 0
>>>> >> }
>>>> >>
>>>> >> ceph status:
>>>> >>
>>>> >> health HEALTH_ERR
>>>> >> 173 pgs are stuck inactive for more than 300 seconds
>>>> >> 173 pgs incomplete
>>>> >> 173 pgs stuck inactive
>>>> >> 173 pgs stuck unclean
>>>> >> monmap e2: 5 mons at
>>>> >>
>>>> >> {ca-cn1=10.50.5.117:6789/0,ca-cn2=10.50.5.118:6789/0,ca-cn3= }10.50.5.119:6789/0,ca-cn4=10.5 0.5.120:6789/0,ca-cn5=10.50.5. 121:6789/0
>>>> >> election epoch 106, quorum 0,1,2,3,4
>>>> >> ca-cn1,ca-cn2,ca-cn3,ca-cn4,ca-cn5
>>>> >> mgr active: ca-cn1 standbys: ca-cn2, ca-cn4, ca-cn5, ca-cn3
>>>> >> osdmap e1128: 60 osds: 59 up, 59 in; 173 remapped pgs
>>>> >> flags sortbitwise,require_jewel_osds,require_kraken_osds
>>>> >> pgmap v782747: 2048 pgs, 1 pools, 63133 GB data, 46293 kobjects
>>>> >> 85199 GB used, 238 TB / 322 TB avail
>>>> >> 1868 active+clean
>>>> >> 173 remapped+incomplete
>>>> >> 7 active+clean+scrubbing
>>>> >>
>>>> >> MON log:
>>>> >>
>>>> >> 2017-01-20 09:25:54.715684 7f55bcafb700 0 log_channel(cluster) log
>>>> >> [INF] :
>>>> >> osd.54 out (down for 31.703786)
>>>> >> 2017-01-20 09:25:54.725688 7f55bf4d5700 0 mon.ca-cn1@0(leader).osd
>>>> >> e1120
>>>> >> crush map has features 288250512065953792, adjusting msgr requires
>>>> >> 2017-01-20 09:25:54.729019 7f55bf4d5700 0 log_channel(cluster) log
>>>> >> [INF] :
>>>> >> osdmap e1120: 60 osds: 59 up, 59 in
>>>> >> 2017-01-20 09:25:54.735987 7f55bf4d5700 0 log_channel(cluster) log
>>>> >> [INF] :
>>>> >> pgmap v781993: 2048 pgs: 1869 active+clean, 173 incomplete, 6
>>>> >> active+clean+scrubbing; 63159 GB data, 85201 GB used, 238 TB / 322 TB
>>>> >> avail;
>>>> >> 21825 B/s rd, 163 MB/s wr, 2046 op/s
>>>> >> 2017-01-20 09:25:55.737749 7f55bf4d5700 0 mon.ca-cn1@0(leader).osd
>>>> >> e1121
>>>> >> crush map has features 288250512065953792, adjusting msgr requires
>>>> >> 2017-01-20 09:25:55.744338 7f55bf4d5700 0 log_channel(cluster) log
>>>> >> [INF] :
>>>> >> osdmap e1121: 60 osds: 59 up, 59 in
>>>> >> 2017-01-20 09:25:55.749616 7f55bf4d5700 0 log_channel(cluster) log
>>>> >> [INF] :
>>>> >> pgmap v781994: 2048 pgs: 29 remapped+incomplete, 1869 active+clean,
>>>> >> 144
>>>> >> incomplete, 6 active+clean+scrubbing; 63159 GB data, 85201 GB used,
>>>> >> 238 TB /
>>>> >> 322 TB avail; 44503 B/s rd, 45681 kB/s wr, 518 op/s
>>>> >> 2017-01-20 09:25:56.768721 7f55bf4d5700 0 log_channel(cluster) log
>>>> >> [INF] :
>>>> >> pgmap v781995: 2048 pgs: 47 remapped+incomplete, 1869 active+clean,
>>>> >> 126
>>>> >> incomplete, 6 active+clean+scrubbing; 63159 GB data, 85201 GB used,
>>>> >> 238 TB /
>>>> >> 322 TB avail; 20275 B/s rd, 72742 kB/s wr, 665 op/s
>>>> >>
>>>> >> Thanks,
>>>> >> Muthu
>>>> >>
>>>> >>
>>>> >> _______________________________________________
>>>> >> ceph-users mailing list
>>>> >> ceph-users@xxxxxxxxxxxxxx
>>>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>> >>
>>>> > _______________________________________________
>>>> > ceph-users mailing list
>>>> > ceph-users@xxxxxxxxxxxxxx
>>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
