Re: ceph infernalis pg creating forever

[German Anders <ganders@xxxxxxxxxxxx>]

Here's the actual crush map:

$ cat /home/ceph/actual_map.out

# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable straw_calc_version 1

# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3
device 4 osd.4
device 5 osd.5
device 6 osd.6
device 7 osd.7
device 8 osd.8
device 9 osd.9
device 10 osd.10
device 11 osd.11
device 12 osd.12
device 13 osd.13
device 14 osd.14
device 15 osd.15
device 16 osd.16
device 17 osd.17
device 18 osd.18
device 19 osd.19
device 20 osd.20
device 21 osd.21
device 22 osd.22
device 23 osd.23
device 24 osd.24
device 25 osd.25
device 26 osd.26
device 27 osd.27
device 28 osd.28
device 29 osd.29
device 30 osd.30
device 31 osd.31

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
host cibn05 {
    id -2        # do not change unnecessarily
    # weight 5.780
    alg straw
    hash 0    # rjenkins1
    item osd.0 weight 0.722
    item osd.1 weight 0.722
    item osd.2 weight 0.722
    item osd.3 weight 0.722
    item osd.4 weight 0.722
    item osd.5 weight 0.722
    item osd.6 weight 0.722
    item osd.7 weight 0.722
}
host cibn06 {
    id -3        # do not change unnecessarily
    # weight 5.780
    alg straw
    hash 0    # rjenkins1
    item osd.8 weight 0.722
    item osd.9 weight 0.722
    item osd.10 weight 0.722
    item osd.11 weight 0.722
    item osd.12 weight 0.722
    item osd.13 weight 0.722
    item osd.14 weight 0.722
    item osd.15 weight 0.722
}
host cibn07 {
    id -4        # do not change unnecessarily
    # weight 5.780
    alg straw
    hash 0    # rjenkins1
    item osd.16 weight 0.722
    item osd.17 weight 0.722
    item osd.18 weight 0.722
    item osd.19 weight 0.722
    item osd.20 weight 0.722
    item osd.21 weight 0.722
    item osd.22 weight 0.722
    item osd.23 weight 0.722
}
host cibn08 {
    id -5        # do not change unnecessarily
    # weight 5.780
    alg straw
    hash 0    # rjenkins1
    item osd.24 weight 0.722
    item osd.25 weight 0.722
    item osd.26 weight 0.722
    item osd.27 weight 0.722
    item osd.28 weight 0.722
    item osd.29 weight 0.722
    item osd.30 weight 0.722
    item osd.31 weight 0.722
}
root default {
    id -1        # do not change unnecessarily
    # weight 23.120
    alg straw
    hash 0    # rjenkins1
    item cibn05 weight 5.780
    item cibn06 weight 5.780
    item cibn07 weight 5.780
    item cibn08 weight 5.780
}

# rules
rule replicated_ruleset {
    ruleset 0
    type replicated
    min_size 1
    max_size 10
    step take default
    step chooseleaf firstn 0 type host
    step emit
}

# end crush map

Kernel version of the servers mon and osd is 3.19.0-25-generic

Regards,

German

2015-11-20 12:56 GMT-03:00 Gregory Farnum <gfarnum@xxxxxxxxxx>:

This usually means your crush mapping for the pool in question is unsatisfiable. Check what the rule is doing.

-Greg

On Friday, November 20, 2015, German Anders <ganders@xxxxxxxxxxxx> wrote:

Hi all, I've finished the install of a new ceph cluster with infernalis 9.2.0 release. But I'm getting the following error msg:

$ ceph -w
    cluster 29xxxxxx-3xxx-xxx9-xxx7-xxxxxxxxb8xx
     health HEALTH_WARN
            64 pgs degraded
            64 pgs stale
            64 pgs stuck degraded
            1024 pgs stuck inactive
            64 pgs stuck stale
            1024 pgs stuck unclean
            64 pgs stuck undersized
            64 pgs undersized
            pool rbd pg_num 1024 > pgp_num 64
     monmap e1: 3 mons at {cibm01=172.23.16.1:6789/0,cibm02=172.23.16.2:6789/0,cibm03=172.23.16.3:6789/0}
            election epoch 6, quorum 0,1,2 cibm01,cibm02,cibm03
     osdmap e113: 32 osds: 32 up, 32 in
            flags sortbitwise
      pgmap v1264: 1024 pgs, 1 pools, 0 bytes data, 0 objects
            1344 MB used, 23673 GB / 23675 GB avail
                 960 creating
                  64 stale+undersized+degraded+peered

2015-11-20 10:22:27.947850 mon.0 [INF] pgmap v1264: 1024 pgs: 960 creating, 64 stale+undersized+degraded+peered; 0 bytes data, 1344 MB used, 23673 GB / 23675 GB avail

It seems like it's 'creating' the new pgs... but, I had been in this state for a while, with actually no changes at all. Also the ceph health detail output:

$ ceph health
HEALTH_WARN 64 pgs degraded; 64 pgs stale; 64 pgs stuck degraded; 1024 pgs stuck inactive; 64 pgs stuck stale; 1024 pgs stuck unclean; 64 pgs stuck undersized; 64 pgs undersized; pool rbd pg_num 1024 > pgp_num 64

$ ceph health detail
(...)
pg 0.31 is stuck stale for 85287.493086, current state stale+undersized+degraded+peered, last acting [0]
pg 0.32 is stuck stale for 85287.493090, current state stale+undersized+degraded+peered, last acting [0]
pg 0.33 is stuck stale for 85287.493093, current state stale+undersized+degraded+peered, last acting [0]
pg 0.34 is stuck stale for 85287.493097, current state stale+undersized+degraded+peered, last acting [0]
pg 0.35 is stuck stale for 85287.493101, current state stale+undersized+degraded+peered, last acting [0]
pg 0.36 is stuck stale for 85287.493105, current state stale+undersized+degraded+peered, last acting [0]
pg 0.37 is stuck stale for 85287.493110, current state stale+undersized+degraded+peered, last acting [0]
pg 0.38 is stuck stale for 85287.493114, current state stale+undersized+degraded+peered, last acting [0]
pg 0.39 is stuck stale for 85287.493119, current state stale+undersized+degraded+peered, last acting [0]
pg 0.3a is stuck stale for 85287.493123, current state stale+undersized+degraded+peered, last acting [0]
pg 0.3b is stuck stale for 85287.493127, current state stale+undersized+degraded+peered, last acting [0]
pg 0.3c is stuck stale for 85287.493131, current state stale+undersized+degraded+peered, last acting [0]
pg 0.3d is stuck stale for 85287.493135, current state stale+undersized+degraded+peered, last acting [0]
pg 0.3e is stuck stale for 85287.493139, current state stale+undersized+degraded+peered, last acting [0]
pg 0.3f is stuck stale for 85287.493149, current state stale+undersized+degraded+peered, last acting [0]
pool rbd pg_num 1024 > pgp_num 64

If I try to increment the pgp_num it will said:

$ ceph osd pool set rbd pgp_num 1024
Error EBUSY: currently creating pgs, wait

$ ceph osd lspools
0 rbd,

$ ceph osd tree
ID WEIGHT   TYPE NAME       UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 23.11963 root default                                     
-2  5.77991     host cibn05                                  
 0  0.72249         osd.0        up  1.00000          1.00000
 1  0.72249         osd.1        up  1.00000          1.00000
 2  0.72249         osd.2        up  1.00000          1.00000
 3  0.72249         osd.3        up  1.00000          1.00000
 4  0.72249         osd.4        up  1.00000          1.00000
 5  0.72249         osd.5        up  1.00000          1.00000
 6  0.72249         osd.6        up  1.00000          1.00000
 7  0.72249         osd.7        up  1.00000          1.00000
-3  5.77991     host cibn06                                  
 8  0.72249         osd.8        up  1.00000          1.00000
 9  0.72249         osd.9        up  1.00000          1.00000
10  0.72249         osd.10       up  1.00000          1.00000
11  0.72249         osd.11       up  1.00000          1.00000
12  0.72249         osd.12       up  1.00000          1.00000
13  0.72249         osd.13       up  1.00000          1.00000
14  0.72249         osd.14       up  1.00000          1.00000
15  0.72249         osd.15       up  1.00000          1.00000
-4  5.77991     host cibn07                                  
16  0.72249         osd.16       up  1.00000          1.00000
17  0.72249         osd.17       up  1.00000          1.00000
18  0.72249         osd.18       up  1.00000          1.00000
19  0.72249         osd.19       up  1.00000          1.00000
20  0.72249         osd.20       up  1.00000          1.00000
21  0.72249         osd.21       up  1.00000          1.00000
22  0.72249         osd.22       up  1.00000          1.00000
23  0.72249         osd.23       up  1.00000          1.00000
-5  5.77991     host cibn08                                  
24  0.72249         osd.24       up  1.00000          1.00000
25  0.72249         osd.25       up  1.00000          1.00000
26  0.72249         osd.26       up  1.00000          1.00000
27  0.72249         osd.27       up  1.00000          1.00000
28  0.72249         osd.28       up  1.00000          1.00000
29  0.72249         osd.29       up  1.00000          1.00000
30  0.72249         osd.30       up  1.00000          1.00000
31  0.72249         osd.31       up  1.00000          1.00000

$ ceph pg dump
(...)
pool 0    0    0    0    0    0    0    0    0
 sum    0    0    0    0    0    0    0    0
osdstat    kbused    kbavail    kb    hb in    hb out
31    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,30]    []
30    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,29]    []
29    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,28]    []
28    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,27]    []
27    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,26]    []
26    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,25]    []
25    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,24]    []
24    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,23]    []
23    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,22]    []
22    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,21]    []
9    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8]    []
8    36896    775752376    775789272    [0,1,2,3,4,5,6,7]    []
7    36896    775752376    775789272    [0,3,5,6]    []
6    36896    775752376    775789272    [0,3,5,7]    []
5    36896    775752376    775789272    [0,3,6,7]    []
4    36896    775752376    775789272    [0,1,2,3,5,6,7]    []
3    36896    775752376    775789272    [0,5,6,7]    []
2    36896    775752376    775789272    [0,1,3,4,5,6,7]    []
1    36896    775752376    775789272    [0,2,3,4,5,6,7]    []
0    36900    775752372    775789272    [3,5,6,7]    []
10    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,9]    []
11    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,10]    []
12    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,11]    []
13    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,12]    []
14    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,13]    []
15    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,14]    []
16    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,15]    []
17    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,16]    []
18    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,17]    []
19    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,18]    []
20    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,19]    []
21    36896    775752376    775789272    [0,1,2,3,4,5,6,7,8,20]    []
 sum    1180676    24824076028    24825256704

Any ideas?

Thanks in advance,

German

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com