Maybe an instance of https://tracker.ceph.com/issues/46847 ? Nest time you see this problem, you can try the new "repeer" command on affected PGs. The "ceph pg x.y query" as mentioned by Etienne will provide a clue if its due to this bug. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Etienne Menguy <etienne.menguy@xxxxxxxx> Sent: 17 August 2021 10:27:14 To: ceph-users Subject: Re: PGs stuck after replacing OSDs Hi, It’s hard to explain as issue is no longer here, if it happens again “ceph pg x.y query” output could be useful. I don’t think you went too fast or removed too many disks in a single step. As you only have 3 nodes, Ceph should have directly noticed degraded PG and could not do much. You didn’t had to set them out as you removed them from crushmap just after. If you are able to change disk without restarting host, I would advise you to do it one by one. Any osd issue on the 2 others servers will lead to a service outage, as you’ll end with a single copy of some PG. If you have to deal with issues, you’ll prefer to have a cluster with 3% of degraded objects rather than 33%. Also, it will be less impacting for your users as more OSD will be available to handle IO. >From my experience, restarting OSD/cluster is sometimes a fix for strange issues. Étienne > On 17 Aug 2021, at 09:41, Ml Ml <mliebherr99@xxxxxxxxxxxxxx> wrote: > > Hello List, > > I am running Proxmox on top of ceph 14.2.20 on the nodes, replica 3, size 2. > > Last week I wanted to swap the HDDs to SDDs on one node. > > Since i have 3 Nodes with replica 3, size 2 i did the following: > > 1.) cep osd set noout > 2.) Stopped all OSD on that one node > 3.) i set the OSDs to out "ceph osd out" on that node > 4.) I removed/destroyed the OSD > 5.) I physically took the disk/osd out > 6.) I plugged my SSDs in and started to add them as OSDs. > > Recovery was active and running, but some PGs did not serve IO and > where stuck. VMs started to complain about IO problems. Looked like > write was not able to some. > > Looked like I had "pgs stuck" and "slow osds blocking"... > ...but "ceph osd perf" and "iostat -dx 3" showed bored/idle OSDs. > > ...I restarted the OSD which seemed to be involved. Which did not help. > > After 1h or so i started to restart ALL the OSDs one by one in the > whole Cluster. After restarting the last OSD in that cluster on a very > different node, the blocking error went away > and everything seemed to recover smoothly. > > > I wonder what i did wrong. > I did those Steps (1-6) within 5mins (so pretty fast). Maybe I should > have taken more time? > Was it too rough to replace all OSDs on one node? > Should I have replaced it one by one? > > Any hints are welcome. > > Cheers, > Mario > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |