I can't offer a spec off the cuff, but if the LV still exists and you
don't need to change its size, then I'd zap it to remove residual Ceph
info because otherwise the operation will complain and fail.
Having done that, the requirements should be the same as a first-time
construction of an OSD on that LV. Eugen can likely give you the spec
info. I'd have to RTFM.
Tim
On 11/1/24 11:22, Dave Hall wrote:
Tim, Eugen,
So what would a spec file look like for a single OSD that uses a specific
HDD (/dev/sdi) and with WAL/DB on an LV that's 25% of a specific NVMe
drive? Regarding the NVMe, there are 3 other OSDs already using 25% each
of this NVMe for WAL/DB, but I have removed the LV that was used by the
failed OSD. Do I need to pre-create the LV, or will 'ceph orch' do that
for me?
Thanks.
-Dave
--
Dave Hall
Binghamton University
kdhall@xxxxxxxxxxxxxx
On Thu, Oct 31, 2024 at 3:52 PM Tim Holloway <timh@xxxxxxxxxxxxx> wrote:
I migrated from gluster when I found out it's going unsupported shortly.
I'm really not big enough for Ceph proper, but there were only so many
supported distributed filesystems with triple redundancy.
Where I got into trouble was that I started off with Octopus and Octopus
had some teething pains. Like stalling scheduled operations until the
system was clan but the only way to get a clean system was to run the
stalled operations. Pacific cured that for me.
But the docs were and remain somewhat fractured between legacy and
managed services and I managed to get into a real mess there, especially
since I was wildly trying anything to get those stalled fixes to take.
Since then, I've pretty much redefined all my OSDs with fewer but larger
datastores and made them all managed. Now if I could just persuade the
auto-tuner to fix the PG sizes,
I'm in the process of opening a ticket account right now. The fun part
of this is that realistically, older docs need a re-write just as much
as the docs for the current release.
Tim
On 10/31/24 15:39, Eugen Block wrote:
I completely understand your point of view. Our own main cluster is
also a bit "wild" in its OSD layout, that's why its OSDs are
"unmanaged" as well. When we adopted it via cephadm, I started to
create suitable osd specs for all those hosts and OSDs and I gave up.
:-D But since we sometimes also tend to experiment a bit, I rather
have full control over it. That's why we also have
osd_crush_initial_weight = 0, to check the OSD creation before letting
Ceph remap any PGs.
It definitely couldn't hurt to clarify the docs, you can always report
on tracker.ceph.com if you have any improvement ideas.
Zitat von Tim Holloway <timh@xxxxxxxxxxxxx>:
I have been slowly migrating towards spec files as I prefer
declarative management as a rule.
However, I think that we may have a dichotomy in the user base.
On the one hand, users with dozens/hundreds of server/drives of
basically identical character.
On the other, I'm one who's running fewer servers and for historical
reasons they tend to be wildly individualistic and often have blocks
of future-use space reserved for non-ceph storage.
Ceph, left to its own devices (no pun intended) can be quite
enthusiastic about adopting any storage it can find. And that's great
for users in the first category. Which is what the spec information
in the supplied links is emphasizing. But for us lesser creatures who
feel the need to manually control where each OSD and how it's
configured, it's not so simple. I'm fairly certain that there's
documentation on the spec file setup for that sort of stuff in the
online docs, but it's located somewhere else and I cannot recall where.
At any rate I would consider it very important that the different
ways to setup an OSD should explicitly indicate which type of OSD
will be generated in their documentation.
Tim
On 10/31/24 14:28, Eugen Block wrote:
Hi,
the preferred method to deploy OSDs in cephadm managed clusters are
spec files, see this part of the docs [0] for more information. I
would just not use the '--all-available-devices' flag, except in
test clusters, or if you're really sure that this is what you want.
If you use 'ceph orch daemon add osd ...', you'll end up with one
(or more) OSD(s), but they will be unmanaged, as you already noted
in your own cluster. There are a couple of examples with advanced
specs (e. g. DB/WAL on dedicated devices) in the docs as well [1].
So my recommendation would be to have a suiting spec file for your
disk layout. You can always check with the '--dry-run' flag before
actually applying it:
ceph orch apply -i osd-spec.yaml --dry-run
Regards,
Eugen
[0] https://docs.ceph.com/en/latest/cephadm/services/osd/#deploy-osds
[1]
https://docs.ceph.com/en/latest/cephadm/services/osd/#advanced-osd-service-specifications
Zitat von Tim Holloway <timh@xxxxxxxxxxxxx>:
As I understand it, the manual OSD setup is only for legacy
(non-container) OSDs. Directory locations are wrong for managed
(containerized) OSDs, for one.
Actually, the whole manual setup docs ought to be moved out of the
mainline documentation. In their present arrangement, they make
legacy setup sound like the preferred method. And have you noticed
that there is no corresponding well-marked section titled
"Authomated (cephadmin) setup?".
This is how we end up with OSDs that are simultaneously legacy AND
administered for the same OSD, since at last count there are no
interlocks within Ceph to prevent such a mess.
Tim
On 10/31/24 13:39, Dave Hall wrote:
Hello.
Sorry if it appears that I am reposting the same issue under a
different
topic. However, I feel that the problem has moved and I now have
different
questions.
At this point I have, I believe, removed all traces of OSD.12 from my
cluster - based on steps in the Reef docs at
https://docs.ceph.com/en/reef/rados/operations/add-or-rm-osds/#.
I have
further located and removed the WAL/DB LV on an associated NVMe drive
(shared with 3 other OSDs).
I don't believe the instructions for replacing an OSD (ceph-volume
lvm
prepare) still apply, so I have been trying to work with the
instructions
under ADDING AN OSD (MANUAL).
However, since my installation is containerized (Podman), it is
unclear
which steps should be issued on the host and which within 'cephadm
shell'.
There is also another ambiguity: In step 3 the instruction is to
'mkfs -t
{fstype}' and then to 'mount -o user_xattr'. However, which fs type?
After this, in step 4, the 'ceph-osd -i {osd-id} --mkfs --mkkey' gets
throws errors about the keyring file.
So, are these the right instructions to be using in a containerized
installation? Are there, in general, alternate documents for
containerized
installations?
Lastly, the above cited instructions don't say anything about the
separate
WAL/DB LV.
Please advise.
Thanks.
-Dave
--
Dave Hall
Binghamton University
kdhall@xxxxxxxxxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
