It's not clear to me if you wanted to add some more details after "I
see this:" (twice).
So you do see backfilling traffic if you out the OSD? Then maybe the
remapped PGs are not even on that OSD? Have you checked 'ceph pg ls
remapped'?
To drain an OSD, you can either set it "out" as you already tried, or
set its crush weight to 0:
ceph osd crush reweight osd.0 0
Zitat von Roland Giesler <roland@xxxxxxxxxxxxxx>:
On 2024/11/14 09:37, Eugen Block wrote:
Remapped PGs is exactly what to expect after removing (or adding) a
device class. Did you revert the change entirely? It sounds like
you maybe forgot to add the original device class back to the OSD
where you changed it? Maybe share 'ceph osd tree'? Do you have
recovery IO (ceph -s)? Does the number of misplaced objects change?
NodeA:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 49.82324 root default
-3 12.36960 host FT1-NodeA
2 hdd 1.86029 osd.2 up 1.00000 1.00000
3 hdd 1.86029 osd.3 up 1.00000 1.00000
4 hdd 1.86029 osd.4 up 1.00000 1.00000
5 hdd 1.86589 osd.5 up 1.00000 1.00000
0 ssd 0.28319 osd.0 up 1.00000 1.00000
1 ssd 0.28319 osd.1 up 1.00000 1.00000
36 ssd 0.20000 osd.36 up 1.00000 1.00000
37 ssd 0.28319 osd.37 up 1.00000 1.00000
38 ssd 0.28319 osd.38 up 1.00000 1.00000
39 ssd 0.28319 osd.39 up 1.00000 1.00000
40 ssd 3.30690 osd.40 up 1.00000 1.00000
-7 12.63338 host FT1-NodeB
10 hdd 1.86029 osd.10 up 1.00000 1.00000
11 hdd 1.86029 osd.11 up 1.00000 1.00000
26 hdd 1.86029 osd.26 up 1.00000 1.00000
27 hdd 1.86029 osd.27 up 1.00000 1.00000
6 ssd 0.28319 osd.6 up 1.00000 1.00000
7 ssd 0.28319 osd.7 up 1.00000 1.00000
8 ssd 0.28319 osd.8 up 1.00000 1.00000
9 ssd 0.28319 osd.9 up 1.00000 1.00000
24 ssd 0.28319 osd.24 up 1.00000 1.00000
25 ssd 0.28319 osd.25 up 1.00000 1.00000
41 ssd 3.49309 osd.41 up 1.00000 1.00000
-10 12.18689 host FT1-NodeC
14 hdd 1.59999 osd.14 up 1.00000 1.00000
15 hdd 1.86029 osd.15 up 1.00000 1.00000
16 hdd 1.86029 osd.16 up 1.00000 1.00000
17 hdd 1.86029 osd.17 up 1.00000 1.00000
12 ssd 0.28319 osd.12 up 1.00000 1.00000
13 ssd 0.28319 osd.13 up 1.00000 1.00000
28 ssd 0.28319 osd.28 up 1.00000 1.00000
29 ssd 0.28319 osd.29 up 1.00000 1.00000
30 ssd 0.28319 osd.30 up 1.00000 1.00000
31 ssd 0.28319 osd.31 up 1.00000 1.00000
43 ssd 3.30690 osd.43 up 1.00000 1.00000
-13 12.63338 host FT1-NodeD
20 hdd 1.86029 osd.20 up 1.00000 1.00000
21 hdd 1.86029 osd.21 up 1.00000 1.00000
22 hdd 1.86029 osd.22 up 1.00000 1.00000
23 hdd 1.86029 osd.23 up 1.00000 1.00000
18 ssd 0.28319 osd.18 up 1.00000 1.00000
19 ssd 0.28319 osd.19 up 1.00000 1.00000
32 ssd 0.28319 osd.32 up 1.00000 1.00000
33 ssd 0.28319 osd.33 up 1.00000 1.00000
34 ssd 0.28319 osd.34 up 1.00000 1.00000
35 ssd 0.28319 osd.35 up 1.00000 1.00000
42 ssd 3.49309 osd.42 up 1.00000 1.00000
No recovery IO, which is what worries me:
NodeA:~# ceph -s
cluster:
id: 04385b88-049f-4083-8d5a-6c45a0b7bddb
health: HEALTH_WARN
1 pool(s) have no replicas configured
services:
mon: 3 daemons, quorum FT1-NodeA,FT1-NodeB,FT1-NodeC (age 3d)
mgr: FT1-NodeC(active, since 3d), standbys: FT1-NodeA, FT1-NodeB
mds: 1/1 daemons up, 1 standby
osd: 44 osds: 44 up (since 14h), 44 in (since 14h); 113 remapped pgs
data:
volumes: 1/1 healthy
pools: 7 pools, 1586 pgs
objects: 5.79M objects, 12 TiB
usage: 24 TiB used, 26 TiB / 50 TiB avail
pgs: 4161/11721328 objects misplaced (0.035%)
1471 active+clean
113 active+clean+remapped
2 active+clean+scrubbing+deep
io:
client: 14 MiB/s rd, 8.5 MiB/s wr, 62 op/s rd, 1.27k op/s wr
progress:
It stays on that "113 active+clean+remapped".
When I "out" osd.0, the status changes for a while (30 minutes
estimated) and then, once the recovery IO stops, when I attempt to
stop the osd, this is displayed:
The 6 "active+remapped+backfilling" pg were many more at first when
i took the osd out of the cluster. If I try to stop the osd now
however, I see this:
Clearly these are more pg's than the 6 that are still backfilling.
Is there a way to force the pg's off this osd, so I safely stop it?
Zitat von Roland Giesler <roland@xxxxxxxxxxxxxx>:
On 2024/11/13 21:05, Anthony D'Atri wrote:
I would think that there was some initial data movement and that
it all went back when you reverted. I would not expect a mess.
data:
volumes: 1/1 healthy
pools: 7 pools, 1586 pgs
objects: 5.79M objects, 12 TiB
usage: 24 TiB used, 26 TiB / 50 TiB avail
pgs: 4161/11720662 objects misplaced (0.036%)
1463 active+clean
113 active+clean+remapped
9 active+clean+scrubbing+deep+repair
1 active+clean+scrubbing+deep
I have 113 active+clean+remapped pg's that just stay like that. If
I try to out the this osd to stop it, the clsuter also never
settles. So then if I try to stop it in the GUI is tells me 73
pg's are still on the OSD...
Can I force those pg's away from the osd?
On Nov 13, 2024, at 12:48 PM, Roland Giesler
<roland@xxxxxxxxxxxxxx> wrote:
I created a new osd class and changed the class of an osd to the
new one without taking the osd out and stopping it first. The
new class also has a crush rule and a pool created for it.
When I realised my mistake, I reverted to what I had before.
However, I suspect that I now have a mess on that osd.
What happened when I did this?
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
