Hi,
during cluster upgrades from L to N or later one had to rebuild OSDs
which were originally deployed by ceph-disk switching to ceph-volume.
We've done this on multiple clusters and redeployed one node by one.
We did not drain the nodes beforehand because the EC resiliency
configuration was well planned. So we also didn't set any flags
because tearing down OSDs only took a few minutes, we didn't care if a
few MB or GB had been recovered. You just need to be careful with the
mon_max_pg_per_osd limit (default 250) as this can easily be hit with
the first rebuilt OSD that's coming up, or the last one being removed
as well. So you could temporarily increase it during the rebuilt
procedure. In your case you probably should set flags because you'll
need time to exchange the physical disks, I guess the ones you already
mentioned would suffice. With stretch mode and replicated size 4 you
should also be able to tear down an entire node at a time and rebuilt
it, I wouldn't rebuild more than one though.
I have not done this yet with cephadm clusters but if your
drivegroup.yml worked correctly before it should work here as well,
hoping that you don't find another ceph-volume bug. ;-)
Regards,
Eugen
Zitat von Zoran Bošnjak <zoran.bosnjak@xxxxxx>:
Hello ceph users,
my ceph configuration is
- ceph version 17.2.5 on ubuntu 20.04
- stretch mode
- 2 rooms with OSDs and monitors + additional room for the tiebreaker monitor
- 4 OSD servers in each room
- 6 OSDs per OSD server
- ceph installation/administration is manual (without ansible,
orch... or any other tool like this)
Ceph health is currently OK.
Raw usage is around 60%,
Pools usage is below 75%
I need to replace all OSD disks in the cluster with larger capacity
disks (500G to 1000G). So the eventual configuration will contain
the same number of OSDs and servers.
I understand I can replace OSDs one by one, following the documented
procedure (removing old and adding new OSD to the configuration) and
waiting for health OK. But in this case, ceph will probably copy
data around like crazy after each step. So, my question is:
What is the recommended procedure in this case of replacing ALL
disks and keeping the ceph operational during the upgrade?
In particular:
Should I use any of "nobackfill, norebalance, norecover..." flags
during the process? If yes, which?
Should I do one OSD at the time, server at the time or even room at the time?
Thanks for the suggestions.
regards,
Zoran
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
