Bob,
Thanks for explanation, sounds resonable! But how it could happen that
host is down and its OSDs are still IN cluster?
I mean NOOUT flag is not set and my timeouts are fully default...
But if I remember correctly host was not completely down, it was
pingable but not other services were reachable like SSH or any others.
Is it possible that OSDs were still sending some information to
monitors making them look like IN?
2015-11-29 2:10 GMT+08:00 Bob R <bobr@xxxxxxxxxxxxxx>:
> Vasiliy,
>
> Your OSDs are marked as 'down' but 'in'.
>
> "Ceph OSDs have two known states that can be combined. Up and Down only
> tells you whether the OSD is actively involved in the cluster. OSD states
> also are expressed in terms of cluster replication: In and Out. Only when a
> Ceph OSD is tagged as Out does the self-healing process occur"
>
> Bob
>
> On Fri, Nov 27, 2015 at 6:15 AM, Mart van Santen <mart@xxxxxxxxxxxx> wrote:
>>
>>
>> Dear Vasilily,
>>
>>
>>
>> On 11/27/2015 02:00 PM, Irek Fasikhov wrote:
>>
>> You have time to synchronize?
>>
>> С уважением, Фасихов Ирек Нургаязович
>> Моб.: +79229045757
>>
>> 2015-11-27 15:57 GMT+03:00 Vasiliy Angapov <angapov@xxxxxxxxx>:
>>>
>>> > It seams that you played around with crushmap, and done something
>>> > wrong.
>>> > Compare the look of 'ceph osd tree' and crushmap. There are some 'osd'
>>> > devices renamed to 'device' think threre is you problem.
>>> Is this a mistake actually? What I did is removed a bunch of OSDs from
>>> my cluster that's why the numeration is sparse. But is it an issue to
>>> a have a sparse numeration of OSDs?
>>
>>
>> I think this is normal and should be no problem. I had this also
>> previously.
>>
>>>
>>> > Hi.
>>> > Vasiliy, Yes it is a problem with crusmap. Look at height:
>>> > -3 14.56000 host slpeah001
>>> > -2 14.56000 host slpeah002
>>> What exactly is wrong here?
>>
>>
>> I do not know how the weight of the hosts contribute to determine were to
>> store the 3-th copy of the PG. As you explained, you have enough space on
>> all hosts, but maybe if the weights of the hosts do not count up and the
>> crushmap maybe come to the conclusion it is not able to place the PGs. What
>> you can try, is to artificially raise the weights of these hosts, to see if
>> it starts mapping the thirth copies for the pg's onto the available host.
>>
>> I had a similiar problem in the past, this was solved by upgrading to the
>> latest crush tunables. But be aware, that can create massive datamovement
>> behavior.
>>
>>
>>>
>>> I also found out that my OSD logs are full of such records:
>>> 2015-11-26 08:31:19.273268 7fe4f49b1700 0 cephx: verify_authorizer
>>> could not get service secret for service osd secret_id=2924
>>> 2015-11-26 08:31:19.273276 7fe4f49b1700 0 --
>>> 192.168.254.18:6816/110740 >> 192.168.254.12:0/1011754 pipe(0x41fd1000
>>> sd=79 :6816 s=0 pgs=0 cs=0 l=1 c=0x3ee1a520).accept: got bad
>>> authorizer
>>> 2015-11-26 08:31:24.273207 7fe4f49b1700 0 auth: could not find
>>> secret_id=2924
>>> 2015-11-26 08:31:24.273225 7fe4f49b1700 0 cephx: verify_authorizer
>>> could not get service secret for service osd secret_id=2924
>>> 2015-11-26 08:31:24.273231 7fe4f49b1700 0 --
>>> 192.168.254.18:6816/110740 >> 192.168.254.12:0/1011754 pipe(0x3f90b000
>>> sd=79 :6816 s=0 pgs=0 cs=0 l=1 c=0x3ee1a3c0).accept: got bad
>>> authorizer
>>> 2015-11-26 08:31:29.273199 7fe4f49b1700 0 auth: could not find
>>> secret_id=2924
>>> 2015-11-26 08:31:29.273215 7fe4f49b1700 0 cephx: verify_authorizer
>>> could not get service secret for service osd secret_id=2924
>>> 2015-11-26 08:31:29.273222 7fe4f49b1700 0 --
>>> 192.168.254.18:6816/110740 >> 192.168.254.12:0/1011754 pipe(0x41fd1000
>>> sd=79 :6816 s=0 pgs=0 cs=0 l=1 c=0x3ee1a260).accept: got bad
>>> authorizer
>>> 2015-11-26 08:31:34.273469 7fe4f49b1700 0 auth: could not find
>>> secret_id=2924
>>> 2015-11-26 08:31:34.273482 7fe4f49b1700 0 cephx: verify_authorizer
>>> could not get service secret for service osd secret_id=2924
>>> 2015-11-26 08:31:34.273486 7fe4f49b1700 0 --
>>> 192.168.254.18:6816/110740 >> 192.168.254.12:0/1011754 pipe(0x3f90b000
>>> sd=79 :6816 s=0 pgs=0 cs=0 l=1 c=0x3ee1a100).accept: got bad
>>> authorizer
>>> 2015-11-26 08:31:39.273310 7fe4f49b1700 0 auth: could not find
>>> secret_id=2924
>>> 2015-11-26 08:31:39.273331 7fe4f49b1700 0 cephx: verify_authorizer
>>> could not get service secret for service osd secret_id=2924
>>> 2015-11-26 08:31:39.273342 7fe4f49b1700 0 --
>>> 192.168.254.18:6816/110740 >> 192.168.254.12:0/1011754 pipe(0x41fcc000
>>> sd=98 :6816 s=0 pgs=0 cs=0 l=1 c=0x3ee19fa0).accept: got bad
>>> authorizer
>>> 2015-11-26 08:31:44.273753 7fe4f49b1700 0 auth: could not find
>>> secret_id=2924
>>> 2015-11-26 08:31:44.273769 7fe4f49b1700 0 cephx: verify_authorizer
>>> could not get service secret for service osd secret_id=2924
>>> 2015-11-26 08:31:44.273776 7fe4f49b1700 0 --
>>> 192.168.254.18:6816/110740 >> 192.168.254.12:0/1011754 pipe(0x41fcc000
>>> sd=98 :6816 s=0 pgs=0 cs=0 l=1 c=0x3ee189a0).accept: got bad
>>> authorizer
>>> 2015-11-26 08:31:49.273412 7fe4f49b1700 0 auth: could not find
>>> secret_id=2924
>>> 2015-11-26 08:31:49.273431 7fe4f49b1700 0 cephx: verify_authorizer
>>> could not get service secret for service osd secret_id=2924
>>> 2015-11-26 08:31:49.273455 7fe4f49b1700 0 --
>>> 192.168.254.18:6816/110740 >> 192.168.254.12:0/1011754 pipe(0x41fd1000
>>> sd=98 :6816 s=0 pgs=0 cs=0 l=1 c=0x3ee19080).accept: got bad
>>> authorizer
>>> 2015-11-26 08:31:54.273293 7fe4f49b1700 0 auth: could not find
>>> secret_id=2924
>>>
>>> What does it mean? Google sais it might be a time sync issue, but my
>>> clocks are perfectly synchronized...
>>
>>
>> Normally you get an error warning in "ceph status" if time is out of sync.
>> Nevertheless, you can try to restart the OSD's. I had issues with timing in
>> the past and discovered it sometime helps to restart the daemons *after*
>> syncing the times, before the accepted the new timings. But this was mostly
>> the case with monitors though.
>>
>>
>>
>> Regards,
>>
>>
>> Mart
>>
>>
>>
>>
>>>
>>> 2015-11-26 21:05 GMT+08:00 Irek Fasikhov <malmyzh@xxxxxxxxx>:
>>> > Hi.
>>> > Vasiliy, Yes it is a problem with crusmap. Look at height:
>>> > " -3 14.56000 host slpeah001
>>> > -2 14.56000 host slpeah002
>>> > "
>>> >
>>> > С уважением, Фасихов Ирек Нургаязович
>>> > Моб.: +79229045757
>>> >
>>> > 2015-11-26 13:16 GMT+03:00 ЦИТ РТ-Курамшин Камиль Фидаилевич
>>> > <Kamil.Kuramshin@xxxxxxxx>:
>>> >>
>>> >> It seams that you played around with crushmap, and done something
>>> >> wrong.
>>> >> Compare the look of 'ceph osd tree' and crushmap. There are some 'osd'
>>> >> devices renamed to 'device' think threre is you problem.
>>> >>
>>> >> Отправлено с мобильного устройства.
>>> >>
>>> >>
>>> >> -----Original Message-----
>>> >> From: Vasiliy Angapov <angapov@xxxxxxxxx>
>>> >> To: ceph-users <ceph-users@xxxxxxxxxxxxxx>
>>> >> Sent: чт, 26 нояб. 2015 7:53
>>> >> Subject: Undersized pgs problem
>>> >>
>>> >> Hi, colleagues!
>>> >>
>>> >> I have small 4-node CEPH cluster (0.94.2), all pools have size 3,
>>> >> min_size
>>> >> 1.
>>> >> This night one host failed and cluster was unable to rebalance saying
>>> >> there are a lot of undersized pgs.
>>> >>
>>> >> root@slpeah002:[~]:# ceph -s
>>> >> cluster 78eef61a-3e9c-447c-a3ec-ce84c617d728
>>> >> health HEALTH_WARN
>>> >> 1486 pgs degraded
>>> >> 1486 pgs stuck degraded
>>> >> 2257 pgs stuck unclean
>>> >> 1486 pgs stuck undersized
>>> >> 1486 pgs undersized
>>> >> recovery 80429/555185 objects degraded (14.487%)
>>> >> recovery 40079/555185 objects misplaced (7.219%)
>>> >> 4/20 in osds are down
>>> >> 1 mons down, quorum 1,2 slpeah002,slpeah007
>>> >> monmap e7: 3 mons at
>>> >>
>>> >>
>>> >> {slpeah001=192.168.254.11:6780/0,slpeah002=192.168.254.12:6780/0,slpeah007=172.31.252.46:6789/0}
>>> >> election epoch 710, quorum 1,2 slpeah002,slpeah007
>>> >> osdmap e14062: 20 osds: 16 up, 20 in; 771 remapped pgs
>>> >> pgmap v7021316: 4160 pgs, 5 pools, 1045 GB data, 180 kobjects
>>> >> 3366 GB used, 93471 GB / 96838 GB avail
>>> >> 80429/555185 objects degraded (14.487%)
>>> >> 40079/555185 objects misplaced (7.219%)
>>> >> 1903 active+clean
>>> >> 1486 active+undersized+degraded
>>> >> 771 active+remapped
>>> >> client io 0 B/s rd, 246 kB/s wr, 67 op/s
>>> >>
>>> >> root@slpeah002:[~]:# ceph osd tree
>>> >> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
>>> >> -1 94.63998 root default
>>> >> -9 32.75999 host slpeah007
>>> >> 72 5.45999 osd.72 up 1.00000 1.00000
>>> >> 73 5.45999 osd.73 up 1.00000 1.00000
>>> >> 74 5.45999 osd.74 up 1.00000 1.00000
>>> >> 75 5.45999 osd.75 up 1.00000 1.00000
>>> >> 76 5.45999 osd.76 up 1.00000 1.00000
>>> >> 77 5.45999 osd.77 up 1.00000 1.00000
>>> >> -10 32.75999 host slpeah008
>>> >> 78 5.45999 osd.78 up 1.00000 1.00000
>>> >> 79 5.45999 osd.79 up 1.00000 1.00000
>>> >> 80 5.45999 osd.80 up 1.00000 1.00000
>>> >> 81 5.45999 osd.81 up 1.00000 1.00000
>>> >> 82 5.45999 osd.82 up 1.00000 1.00000
>>> >> 83 5.45999 osd.83 up 1.00000 1.00000
>>> >> -3 14.56000 host slpeah001
>>> >> 1 3.64000 osd.1 down 1.00000 1.00000
>>> >> 33 3.64000 osd.33 down 1.00000 1.00000
>>> >> 34 3.64000 osd.34 down 1.00000 1.00000
>>> >> 35 3.64000 osd.35 down 1.00000 1.00000
>>> >> -2 14.56000 host slpeah002
>>> >> 0 3.64000 osd.0 up 1.00000 1.00000
>>> >> 36 3.64000 osd.36 up 1.00000 1.00000
>>> >> 37 3.64000 osd.37 up 1.00000 1.00000
>>> >> 38 3.64000 osd.38 up 1.00000 1.00000
>>> >>
>>> >> Crushmap:
>>> >>
>>> >> # begin crush map
>>> >> tunable choose_local_tries 0
>>> >> tunable choose_local_fallback_tries 0
>>> >> tunable choose_total_tries 50
>>> >> tunable chooseleaf_descend_once 1
>>> >> tunable chooseleaf_vary_r 1
>>> >> tunable straw_calc_version 1
>>> >> tunable allowed_bucket_algs 54
>>> >>
>>> >> # devices
>>> >> device 0 osd.0
>>> >> device 1 osd.1
>>> >> device 2 device2
>>> >> device 3 device3
>>> >> device 4 device4
>>> >> device 5 device5
>>> >> device 6 device6
>>> >> device 7 device7
>>> >> device 8 device8
>>> >> device 9 device9
>>> >> device 10 device10
>>> >> device 11 device11
>>> >> device 12 device12
>>> >> device 13 device13
>>> >> device 14 device14
>>> >> device 15 device15
>>> >> device 16 device16
>>> >> device 17 device17
>>> >> device 18 device18
>>> >> device 19 device19
>>> >> device 20 device20
>>> >> device 21 device21
>>> >> device 22 device22
>>> >> device 23 device23
>>> >> device 24 device24
>>> >> device 25 device25
>>> >> device 26 device26
>>> >> device 27 device27
>>> >> device 28 device28
>>> >> device 29 device29
>>> >> device 30 device30
>>> >> device 31 device31
>>> >> device 32 device32
>>> >> device 33 osd.33
>>> >> device 34 osd.34
>>> >> device 35 osd.35
>>> >> device 36 osd.36
>>> >> device 37 osd.37
>>> >> device 38 osd.38
>>> >> device 39 device39
>>> >> device 40 device40
>>> >> device 41 device41
>>> >> device 42 device42
>>> >> device 43 device43
>>> >> device 44 device44
>>> >> device 45 device45
>>> >> device 46 device46
>>> >> device 47 device47
>>> >> device 48 device48
>>> >> device 49 device49
>>> >> device 50 device50
>>> >> device 51 device51
>>> >> device 52 device52
>>> >> device 53 device53
>>> >> device 54 device54
>>> >> device 55 device55
>>> >> device 56 device56
>>> >> device 57 device57
>>> >> device 58 device58
>>> >> device 59 device59
>>> >> device 60 device60
>>> >> device 61 device61
>>> >> device 62 device62
>>> >> device 63 device63
>>> >> device 64 device64
>>> >> device 65 device65
>>> >> device 66 device66
>>> >> device 67 device67
>>> >> device 68 device68
>>> >> device 69 device69
>>> >> device 70 device70
>>> >> device 71 device71
>>> >> device 72 osd.72
>>> >> device 73 osd.73
>>> >> device 74 osd.74
>>> >> device 75 osd.75
>>> >> device 76 osd.76
>>> >> device 77 osd.77
>>> >> device 78 osd.78
>>> >> device 79 osd.79
>>> >> device 80 osd.80
>>> >> device 81 osd.81
>>> >> device 82 osd.82
>>> >> device 83 osd.83
>>> >>
>>> >> # types
>>> >> type 0 osd
>>> >> type 1 host
>>> >> type 2 chassis
>>> >> type 3 rack
>>> >> type 4 row
>>> >> type 5 pdu
>>> >> type 6 pod
>>> >> type 7 room
>>> >> type 8 datacenter
>>> >> type 9 region
>>> >> type 10 root
>>> >>
>>> >> # buckets
>>> >> host slpeah007 {
>>> >> id -9 # do not change unnecessarily
>>> >> # weight 32.760
>>> >> alg straw
>>> >> hash 0 # rjenkins1
>>> >> item osd.72 weight 5.460
>>> >> item osd.73 weight 5.460
>>> >> item osd.74 weight 5.460
>>> >> item osd.75 weight 5.460
>>> >> item osd.76 weight 5.460
>>> >> item osd.77 weight 5.460
>>> >> }
>>> >> host slpeah008 {
>>> >> id -10 # do not change unnecessarily
>>> >> # weight 32.760
>>> >> alg straw
>>> >> hash 0 # rjenkins1
>>> >> item osd.78 weight 5.460
>>> >> item osd.79 weight 5.460
>>> >> item osd.80 weight 5.460
>>> >> item osd.81 weight 5.460
>>> >> item osd.82 weight 5.460
>>> >> item osd.83 weight 5.460
>>> >> }
>>> >> host slpeah001 {
>>> >> id -3 # do not change unnecessarily
>>> >> # weight 14.560
>>> >> alg straw
>>> >> hash 0 # rjenkins1
>>> >> item osd.1 weight 3.640
>>> >> item osd.33 weight 3.640
>>> >> item osd.34 weight 3.640
>>> >> item osd.35 weight 3.640
>>> >> }
>>> >> host slpeah002 {
>>> >> id -2 # do not change unnecessarily
>>> >> # weight 14.560
>>> >> alg straw
>>> >> hash 0 # rjenkins1
>>> >> item osd.0 weight 3.640
>>> >> item osd.36 weight 3.640
>>> >> item osd.37 weight 3.640
>>> >> item osd.38 weight 3.640
>>> >> }
>>> >> root default {
>>> >> id -1 # do not change unnecessarily
>>> >> # weight 94.640
>>> >> alg straw
>>> >> hash 0 # rjenkins1
>>> >> item slpeah007 weight 32.760
>>> >> item slpeah008 weight 32.760
>>> >> item slpeah001 weight 14.560
>>> >> item slpeah002 weight 14.560
>>> >> }
>>> >>
>>> >> # rules
>>> >> rule default {
>>> >> ruleset 0
>>> >> type replicated
>>> >> min_size 1
>>> >> max_size 10
>>> >> step take default
>>> >> step chooseleaf firstn 0 type host
>>> >> step emit
>>> >> }
>>> >>
>>> >> # end crush map
>>> >>
>>> >>
>>> >>
>>> >> This is odd because pools have size 3 and I have 3 hosts alive, so why
>>> >> it is saying that undersized pgs are present? It makes me feel like
>>> >> CRUSH is not working properly.
>>> >> There is not much data currently in cluster, something about 3TB and
>>> >> as you can see from osd tree - each host have minimum of 14TB disk
>>> >> space on OSDs.
>>> >> So I'm a bit stuck now...
>>> >> How can I find the source of trouble?
>>> >>
>>> >> Thanks in advance!
>>> >> _______________________________________________
>>> >> ceph-users mailing list
>>> >> ceph-users@xxxxxxxxxxxxxx
>>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >>
>>> >> _______________________________________________
>>> >> ceph-users mailing list
>>> >> ceph-users@xxxxxxxxxxxxxx
>>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >>
>>> >
>>
>>
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>> --
>> Mart van Santen
>> Greenhost
>> E: mart@xxxxxxxxxxxx
>> T: +31 20 4890444
>> W: https://greenhost.nl
>>
>> A PGP signature can be attached to this e-mail,
>> you need PGP software to verify it.
>> My public key is available in keyserver(s)
>> see: http://tinyurl.com/openpgp-manual
>>
>> PGP Fingerprint: CA85 EB11 2B70 042D AF66 B29A 6437 01A1 10A3 D3A5
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
