----- Le 23 Oct 24, à 20:14, Bob Gibson rjg@xxxxxxxxxx a écrit : > Sorry to resurrect this thread, but while I was able to get the cluster healthy > again by manually creating the osd, I'm still unable to manage osds using the > orchestrator. > > The orchestrator is generally working, but It appears to be unable to scan > devices. Immediately after failing out the mgr `ceph orch device ls` will > display device status from >4 weeks ago, which was when we converted the > cluster to be managed by cephadm. Eventually the orchestrator will attempt to > refresh its device status. At this point `ceph orch device ls` stops displaying > any output at all. I can reproduce this state almost immediately if I run `ceph > orch device ls —refresh` to force an immediate refresh. The mgr log shows > events like the following just before `ceph orch device ls` stops reporting > output (one event for every osd node in the cluster): > > "Detected new or changed devices on ceph-osd31” > > Here are the osd services in play: > > # ceph orch ls osd > NAME PORTS RUNNING REFRESHED AGE PLACEMENT > osd 95 8m ago - <unmanaged> > osd.ceph-osd31 4 8m ago 6d ceph-osd31 > > # ceph orch ls osd --export > service_type: osd > service_name: osd > unmanaged: true > spec: > filter_logic: AND > objectstore: bluestore > --- > service_type: osd > service_id: ceph-osd31 > service_name: osd.ceph-osd31 > placement: > hosts: > - ceph-osd31 > spec: > data_devices: > rotational: 0 > size: '3TB:' > encrypted: true > filter_logic: AND > objectstore: bluestore > > I tried deleting the default “osd” service in case it was somehow conflicting > with my per-node spec, but it looks like that’s not allowed, so I assume any > custom osd service specs override the unmanaged default. > > # ceph orch rm osd > Invalid service 'osd'. Use 'ceph orch ls' to list available services. > Hi Bob, I think this message shows up as this very specific post adoption 'osd' service has already been marked as 'deleted'. Maybe when you ran the command for the first time. The only reason it still shows up on 'ceph orch ls' is that 95 OSDs are still referencing this service in their configuration. Once you'll have edited all OSDs /var/lib/ceph/$(ceph fsid)/osd.xxx/unit.meta files (changed their service_name) and restarted all OSDs (or recreated these 95 OSDs encrypted under another service_name), the 'osd' service will disappear by itself and won't show up anymore on 'ceph orch ls' output. At least this is what I've observed in the past. > My hunch is that some persistent state is corrupted, or there’s something else > preventing the orchestrator from successfully refreshing its device status, but > I don’t know how to troubleshoot this. Any ideas? I don't think this is related to the 'osd' service. As suggested by Tobi, enabling cephadm debug will tell you more. Cheers, Frédéric. > > Cheers, > /rjg > > P.S. @Eugen: When I first started this thread you said it was unnecessary to > destroy an osd to convert it from unmanaged to managed. Can you explain how > this is done? Although we want to recreate the osds to enable encryption, it > would save time, and unnecessary wear on the SSDs, while troubleshooting. > > On Oct 16, 2024, at 2:45 PM, Eugen Block <eblock@xxxxxx> wrote: > > EXTERNAL EMAIL | USE CAUTION > > Glad to hear it worked out for you! > > Zitat von Bob Gibson <rjg@xxxxxxxxxx>: > > I’ve been away on vacation and just got back to this. I’m happy to > report that manually recreating the OSD with ceph-volume and then > adopting it with cephadm fixed the problem. > > Thanks again for your help Eugen! > > Cheers, > /rjg > > On Sep 29, 2024, at 10:40 AM, Eugen Block <eblock@xxxxxx> wrote: > > EXTERNAL EMAIL | USE CAUTION > > Okay, apparently this is not what I was facing. I see two other > options right now. The first would be to purge osd.88 from the crush > tree entirely. > The second approach would be to create an osd manually with > ceph-volume, not cephadm ceph-volume, to create a legacy osd (you'd > get warnings about a stray daemon). If that works, adopt the osd with > cephadm. > I don't have a better idea right now. > > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |