Hi everybody,
I am trying to understand why am I keep on getting remapped+wait_backfill pg statuses, when doing some cluster pg shuffling. Sometimes it happens just by doing small reweight-by-utilization operation, and sometimes when I modify the crushmap (bigger movement of data).
Taking look at ceph health detail and investigating some of the pgs by using pg #ipg_id query, I can see that all of the "acting" pgs are healthy and of the same size. "Up" pgs do have a pg folder created, but dont have any data inside (empty head + TEMP).
I dont have any (near)full pgs, and ceph pg debug unfound_objects_exist yields FALSE.
Cluster is also 100% functional(but in WARN state), and I can see that if I write some data, acting pgs are all happily syncing between each other.
Acting pgs
Up pgs
I can simply recover form this following this steps:
- set noout, norecover, norebalance to avoid unnecessary data movement
- stopping all actingbackfill pgs (active + up) at the same time
- remove empty "up" pgs
- start all the pgs again
- unset noout, norecover, norebalance
After that new "up" pgs are recreated in remapped+backfilling state, and marked as active+clean after some time.
I have also tried to "kick the cluster in the head" with ceph pg send_pg_creates (as stated here https://www.mail-archive.com/ceph-devel@xxxxxxxxxxxxxxx/msg12287.html), but I get:
$ ceph pg send_pg_creates
Error EINVAL: (22) Invalid argument
BTW What is send_pg_creates really supposed to do?
Does anyone have some hints is this occurring?
Thank you,
Ivan
Jewel 12.2.2
size 3
min_size 2
#I havent been playing with tunables
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable straw_calc_version 1
#standard ruleset
rule replicated_ruleset {
ruleset 0
type replicated
min_size 1
max_size 10
step take dc
step chooseleaf firstn 0 type host
step emit
}
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
