[root@gnosis ~]# ceph status
cluster:
id:
health: HEALTH_WARN
8283238/3566503213 objects misplaced (0.232%)
1 pools nearfull
services:
mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03
mgr: ceph-02(active), standbys: ceph-03, ceph-01
mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay
osd: 315 osds: 295 up, 295 in; 8 remapped pgs
data:
pools: 11 pools, 3215 pgs
objects: 455.8 M objects, 817 TiB
usage: 1.0 PiB used, 1.0 PiB / 2.1 PiB avail
pgs: 8283238/3566503213 objects misplaced (0.232%)
3200 active+clean
8 active+remapped+backfilling
7 active+clean+scrubbing+deep
io:
client: 85 MiB/s rd, 93 MiB/s wr, 1.35 kop/s rd, 856 op/s wr
recovery: 0 B/s, 57 objects/s
With this rate I will wait for ages.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: David Caro <dcaro@xxxxxxxxxxxxx>
Sent: 03 December 2020 11:03:28
To: Frank Schilder
Cc: Janne Johansson; ceph-users@xxxxxxx
Subject: Re: Re: Increase number of objects in flight during recovery
Hi Frank,
out of curiosity, can you share the recovery rates you are seeing?
I would appreciate it, thanks!
On 12/03 09:44, Frank Schilder wrote:
> Hi Janne,
>
> looked at it already. The recovery rate is unbearably slow and I would like to increase it. The % misplaced objects is decreasing unnecessarily slow.
>
> Best regards,
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> ________________________________________
> From: Janne Johansson <icepic.dz@xxxxxxxxx>
> Sent: 03 December 2020 10:41:29
> To: Frank Schilder
> Cc: ceph-users@xxxxxxx
> Subject: Re: Increase number of objects in flight during recovery
>
> Den tors 3 dec. 2020 kl 10:11 skrev Frank Schilder <frans@xxxxxx<mailto:frans@xxxxxx>>:
> I have the opposite problem as discussed in "slow down keys/s in recovery". I need to increase the number of objects in flight during rebalance. It is already all remapped PGs in state backfilling, but it looks like no more than 8 objects/sec are transferred per PG at a time. The pools sits on high-performance SSDs and could easily handle a transfer of 100 or more objects/sec simultaneously. Is there any way to increase the number of transfers/sec or simultaneous transfers? Increasing the options osd_max_backfills and osd_recovery_max_active has no effect.
> Background: The pool in question (con-fs2-meta2) is the default data pool of a ceph fs, which stores exclusively the kind of meta data that goes into this pool. Storage consumption is reported as 0, but the number of objects is huge:
>
> I don't run cephfs so it might not map 100%, but I think that pools for which ceph stores file/object metadata (radosgw pools in my case) will show a completely "false" numbers while recovering, which I think is because there are tons of object metadata applied as metadata on 0-sized objects. This means recovery will look like it does one object per second or something, while in fact it does 100s of metadatas on that one object but the recovery doesn't list this. Also, it made old ceph df and rados df say "this pool is almost empty" but when you try to dump or move the pool it takes far longer than it should take to move an almost-empty pool. And the pool dump gets huge.
>
> I would take a look at iostat output for those OSD drives and see if there are 8 iops or lots more actually.
>
> --
> May the most significant bit of your life be positive.
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
--
David Caro
SRE - Cloud Services
Wikimedia Foundation <https://wikimediafoundation.org/>
PGP Signature: 7180 83A2 AC8B 314F B4CE 1171 4071 C7E1 D262 69C3
"Imagine a world in which every single human being can freely share in the
sum of all knowledge. That's our commitment."
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
