Hi Brad,
Thank you for the answer.
We are aware of the fact that hammer is close to retirement, and we are planning for the upgrade. BTW: can you recommend some documentation to read before the hammer -> jewel upgrade? I know http://docs.ceph.com/docs/jewel/install/upgrading-ceph/ and that google is my friend, but I'm asking just to be sure I'm not missing something that could be important.
Thank you.
Laszlo
On 31.05.2017 02:51, Brad Hubbard wrote:
It should also be noted that hammer is pretty close to retirement and
is a poor choice for new clusters.
On Wed, May 31, 2017 at 6:17 AM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote:
On Mon, May 29, 2017 at 4:58 AM, Laszlo Budai <laszlo@xxxxxxxxxxxxxxxx> wrote:
Hello all,
We have a ceph cluster with 72 OSDs distributed on 6 hosts, in 3 chassis. In
our crush map the we are distributing the PGs on chassis (complete crush map
below):
# rules
rule replicated_ruleset {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type chassis
step emit
}
We had a host failure, and I can see that ceph is using 2 OSDs from the same
chassis for a lot of the remapped PGs. Even worse, I can see that there are
cases when a PG is using two OSDs from the same host like here:
3.5f6 37 0 4 37 0 149446656 3040 3040
active+remapped 2017-05-26 11:29:23.122820 61820'222074 61820:158025
[52,39] 52 [52,39,3] 52 61488'198356 2017-05-23
23:51:56.210597 61488'198356 2017-05-23 23:51:56.210597
I have tis in the log:
2017-05-26 11:26:53.244424 osd.52 10.12.193.69:6801/7044 1510 : cluster
[INF] 3.5f6 restarting backfill on osd.39 from (0'0,0'0] MAX to 61488'203000
What can be wrong?
It's not clear from the output you've provided whether your pools have
size 2 or 3. From what you've shown, I'm guessing you have size 2, and
the OSD failure prompted a move of the PG in question away from OSD 3
to OSD 39. Since 39 doesn't have any of the data yet, OSD 3 is being
maintained in the acting set to maintain redundancy, but it will go
away one the backfill is done.
In general, it's a failure of CRUSH's design goals if you see moves of
the replica within buckets which didn't experience failure, but they
do sometimes happen. There have been a lot of improvements over the
years to reduce how often that happens, some of which are supported by
Hammer but not on by default (because it prevents use of older
clients), some of which are only in very new code like the Luminous
dev releases. I suspect you'd find things behave better under your
cluster if you upgrade to Jewel and set the CRUSH flags it recommends
to you.
-Greg
Our crush map looks like this:
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable straw_calc_version 1
# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3
....
device 69 osd.69
device 70 osd.70
device 71 osd.71
# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root
# buckets
host tv-c1-al01 {
id -7 # do not change unnecessarily
# weight 21.840
alg straw
hash 0 # rjenkins1
item osd.5 weight 1.820
item osd.11 weight 1.820
item osd.17 weight 1.820
item osd.23 weight 1.820
item osd.29 weight 1.820
item osd.35 weight 1.820
item osd.41 weight 1.820
item osd.47 weight 1.820
item osd.53 weight 1.820
item osd.59 weight 1.820
item osd.65 weight 1.820
item osd.71 weight 1.820
}
host tv-c1-al02 {
id -3 # do not change unnecessarily
# weight 21.840
alg straw
hash 0 # rjenkins1
item osd.1 weight 1.820
item osd.7 weight 1.820
item osd.13 weight 1.820
item osd.19 weight 1.820
item osd.25 weight 1.820
item osd.31 weight 1.820
item osd.37 weight 1.820
item osd.43 weight 1.820
item osd.49 weight 1.820
item osd.55 weight 1.820
item osd.61 weight 1.820
item osd.67 weight 1.820
}
chassis tv-c1 {
id -8 # do not change unnecessarily
# weight 43.680
alg straw
hash 0 # rjenkins1
item tv-c1-al01 weight 21.840
item tv-c1-al02 weight 21.840
}
host tv-c2-al01 {
id -5 # do not change unnecessarily
# weight 21.840
alg straw
hash 0 # rjenkins1
item osd.3 weight 1.820
item osd.9 weight 1.820
item osd.15 weight 1.820
item osd.21 weight 1.820
item osd.27 weight 1.820
item osd.33 weight 1.820
item osd.39 weight 1.820
item osd.45 weight 1.820
item osd.51 weight 1.820
item osd.57 weight 1.820
item osd.63 weight 1.820
item osd.70 weight 1.820
}
host tv-c2-al02 {
id -2 # do not change unnecessarily
# weight 21.840
alg straw
hash 0 # rjenkins1
item osd.0 weight 1.820
item osd.6 weight 1.820
item osd.12 weight 1.820
item osd.18 weight 1.820
item osd.24 weight 1.820
item osd.30 weight 1.820
item osd.36 weight 1.820
item osd.42 weight 1.820
item osd.48 weight 1.820
item osd.54 weight 1.820
item osd.60 weight 1.820
item osd.66 weight 1.820
}
chassis tv-c2 {
id -9 # do not change unnecessarily
# weight 43.680
alg straw
hash 0 # rjenkins1
item tv-c2-al01 weight 21.840
item tv-c2-al02 weight 21.840
}
host tv-c1-al03 {
id -6 # do not change unnecessarily
# weight 21.840
alg straw
hash 0 # rjenkins1
item osd.4 weight 1.820
item osd.10 weight 1.820
item osd.16 weight 1.820
item osd.22 weight 1.820
item osd.28 weight 1.820
item osd.34 weight 1.820
item osd.40 weight 1.820
item osd.46 weight 1.820
item osd.52 weight 1.820
item osd.58 weight 1.820
item osd.64 weight 1.820
item osd.69 weight 1.820
}
host tv-c2-al03 {
id -4 # do not change unnecessarily
# weight 21.840
alg straw
hash 0 # rjenkins1
item osd.2 weight 1.820
item osd.8 weight 1.820
item osd.14 weight 1.820
item osd.20 weight 1.820
item osd.26 weight 1.820
item osd.32 weight 1.820
item osd.38 weight 1.820
item osd.44 weight 1.820
item osd.50 weight 1.820
item osd.56 weight 1.820
item osd.62 weight 1.820
item osd.68 weight 1.820
}
chassis tv-c3 {
id -10 # do not change unnecessarily
# weight 43.680
alg straw
hash 0 # rjenkins1
item tv-c1-al03 weight 21.840
item tv-c2-al03 weight 21.840
}
root default {
id -1 # do not change unnecessarily
# weight 131.040
alg straw
hash 0 # rjenkins1
item tv-c1 weight 43.680
item tv-c2 weight 43.680
item tv-c3 weight 43.680
}
# rules
rule replicated_ruleset {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type chassis
step emit
}
# end crush map
Thank you,
Laszlo
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
