Anything strange with CPU/IO usage? perf or gdbprof might be useful to
see if it's hanging up on anything.
Mark
On 09/08/2017 10:41 AM, Xiaoxi Chen wrote:
Not changing significantly, still like
8192 pgs: 36 active+degraded+remapped+backfill_wait, 89
active+recovery_wait+degraded+remapped, 95
active+undersized+degraded+remapped+backfilling, 2498
active+undersized+degraded
+remapped+backfill_wait, 1159 active+clean, 75
active+remapped+backfilling, 4240 active+remapped+backfill_wait; 5494
GB data, 20554 GB used, 3061 TB / 3082 TB avail; 920 kB/s wr,
163 op/s; 413811/4301073 objects degraded (9.621%); 3126592/4301073
objects misplaced (72.693%); 9832 kB/s, 2 objects/s recovering
2017-09-08 23:29 GMT+08:00 Sage Weil <sage@xxxxxxxxxxxx>:
On Fri, 8 Sep 2017, Xiaoxi Chen wrote:
Hi,
We consistently saw very slow backfilling speed on our cluster,
although max_backfill set to 50 and do have tens of PGs in
backfilling, but the recovery speed still are only tens of MB, even 0
in some time.
The background is, we first have a pool with all bluestore(on
hdd, db also on hdd) hosting ~ 6TB data across 24 OSDs. Then we have
another 24 OSDs with db on SSD. I was trying to migrade the data from
the old 24 OSDs to new 24 OSDs, with DB on SSD.
So what I did is reweight crush weight of all old OSD to 0 and
all new OSDs to 5, obviously almost all object will goes to misplaced
state and PGs are in "active + remaped+ backfilling/wait_backfill".
Everything looks fine except the backfilling speed is extermply low.
See below pg stat output for instance, pls ignore the degraded PG ,
that was due to I manually mark down 2 old OSDs, just to see if it
could change anything, but nothing.
Every 2.0s: ceph pg stat --cluster pre-prod
Fri Sep 8 08:13:10 2017
8192 pgs: 89 undersized+degraded+remapped+backfilling+peered, 324
undersized+degraded+remapped+backfill_wait+peered, 36
active+undersized+degraded+remapped+backfilling, 2595 activ
e+undersized+degraded+remapped+backfill_wait, 1088 active+clean, 46
active+remapped+backfilling, 4014 active+remapped+backfill_wait; 5494
GB data, 18768 GB used, 3047 TB / 3065 TB
avail; 325 kB/s wr, 64 op/s; 539794/4301028 objects degraded
(12.550%); 3025778/4301028 objects misplaced (70.350%); 28794 kB/s, 7
objects/s recovering
Can you try setting
osd_recovery_sleep = 0
on the OSDs and see if that makes a difference?
sage
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
