The one issue I'm seeing and probably the root of my problem is that cephadm set the user 'ceph' uid to 167...it's something else entirely on my system (perhaps from the fact that it's an older Luminous cluster built with ceph-deploy).
However, even when changing the ceph uid to what cephadm/docker is looking for (167), something is changing the perms on /dev/dm-1.
Annnnnd I got it working using the udev rules you provided! So, I think for my whole issue, I'll need to make sure the uid & gid for the ceph user is set to 167 (not sure why that was set but the fix is easy enough) and have udev rules avail to properly set the perms on /dev/dm-X as such.
Thanks!
________________________________________
From: Andre Goree <agoree@xxxxxxxxxxxxxxxxxx>
Sent: Tuesday, December 28, 2021 6:40 PM
To: Mazzystr
Cc: ceph-users@xxxxxxx
Subject: Re: Converting to cephadm from ceph-deploy
Thank you! I did figure that it maybe should be a soft link, and in fact I tried to fix it by linking everything properly, but as you've shown with your 'ls' example of that directory, I certainly missed a few things. This helps immensely.
Oddly enough, however, even the dir '/var/lib/ceph/osd/ceph-X' itself does not exist, and if I'm not mistaken, is copied to '/var/lib/ceph/$FSID/osd-X'. Easy enough to determine how that needs to be symlinked, and inside 'osd-X' I see the relevant 'block' link so it does appear that everything's there. The perms are another aspect I hadn't considered. I'm going to try to work this out and report back, thanks!
________________________________________
From: Mazzystr <mazzystr@xxxxxxxxx>
Sent: Tuesday, December 28, 2021 5:10 PM
To: Andre Goree
Cc: ceph-users@xxxxxxx
Subject: Re: Converting to cephadm from ceph-deploy
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
/var/lib/ceph/osd/ceph-X/block is a soft link. Track down the soft link chain to the devmapper device. Make sure ceph:ceph owns it
Example:
blah:/var/lib/ceph/osd/ceph-0 # ls -la block*
total 44
lrwxrwxrwx 1 ceph ceph 23 Apr 11 2019 block -> /dev/mapper/ceph-0block
lrwxrwxrwx 1 ceph ceph 20 Apr 11 2019 block.db -> /dev/mapper/ceph-0db
lrwxrwxrwx 1 ceph ceph 21 Apr 11 2019 block.wal -> /dev/mapper/ceph-0wal
blah:/var/lib/ceph/osd/ceph-0 # ls -la /dev/mapper/ceph-0block
lrwxrwxrwx 1 root root 8 Dec 28 12:41 /dev/mapper/ceph-0block -> ../dm-30
blah:/var/lib/ceph/osd/ceph-0 # ls -la /dev/dm-30
brw-rw---- 1 ceph ceph 254, 30 Dec 28 14:05 /dev/dm-30
I land a udev rule to the host to help me correct the ownership problem
cat > /etc/udev/rules.d/99-ceph-osd-${OSD_ID}.rules << EOF
ENV{DM_NAME}=="ceph-${OSD_ID}" OWNER="ceph" GROUP="ceph" MODE="0660"
ENV{DM_NAME}=="ceph-${OSD_ID}wal" OWNER="ceph" GROUP="ceph" MODE="0660"
ENV{DM_NAME}=="ceph-${OSD_ID}db" OWNER="ceph" GROUP="ceph" MODE="0660"
ENV{DM_NAME}=="ceph-${OSD_ID}block" OWNER="ceph" GROUP="ceph" MODE="0660"
EOF
On Tue, Dec 28, 2021 at 12:42 PM Andre Goree <agoree@xxxxxxxxxxxxxxxxxx<mailto:agoree@xxxxxxxxxxxxxxxxxx>> wrote:
First off, I made a similar post on 12/11/21 but had not explicitly signed up for the new mailing list (this email is a remnant from when the list was run with mailman) and I didn't get a reply here and couldn't reply, so I have to make this again, I apologize of the noise).
Hello all. I'm upgrading a cluster from (Ubuntu 16.04) Luminous to Pacific, within
which I've upgraded to (18.04) Nautilus, then to (20.04) Octopus. The cluster ran
flawlessly througout that upgrade process which I'm very happy about.
I'm now at the point of converting the cluster to cephadm (it was built with
ceph-deploy), but I'm running into trouble. I've followed this doc:
https://docs.ceph.com/en/latest/cephadm/adoption/
3 MON nodes
4 OSD nodes
The trouble is two-fold: (1) it seems to be that once I've adopted the MON & MGR
daemons, I can't seem to get the localhost MON to list with "ceph orch ps"
only the two other MON nodes:
#### On MON node ####
root@cephmon01test:~# ceph orch ps
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM
VERSION IMAGE ID CONTAINER ID
mgr.cephmon02test cephmon02test running (21h) 8m ago 21h 365M -
16.2.5 6933c2a0b7dd e08de388b92e
mgr.cephmon03test cephmon03test running (21h) 6m ago 21h 411M -
16.2.5 6933c2a0b7dd d358b697e49b
mon.cephmon02test cephmon02test running (21h) 8m ago - 934M 2048M
16.2.5 6933c2a0b7dd f349d7cc6816
mon.cephmon03test cephmon03test running (21h) 6m ago - 923M 2048M
16.2.5 6933c2a0b7dd 64880b0659cc
root@cephmon01test:~# ceph orch ls
NAME PORTS RUNNING REFRESHED AGE PLACEMENT
mgr 2/0 8m ago - <unmanaged>
mon 2/0 8m ago - <unmanaged>
All of the 'cephadm adopt' commands for the MONs and MGRs were run from the above
node.
My second issue is that when I proceed to adopt the OSDs (again, following
https://docs.ceph.com/en/latest/cephadm/adoption/), they seem to drop out of the cluster:
### on OSD node ###
root@cephosd01test:~# cephadm ls
[
{
"style": "cephadm:v1",
"name": "osd.0",
"fsid": "4cfa6467-6647-41e9-8184-1cacc408265c",
"systemd_unit":
"ceph-4cfa6467-6647-41e9-8184-1cacc408265c(a)osd.0"sd.0",
"enabled": true,
"state": "error",
"container_id": null,
"container_image_name": "ceph/ceph:v16",
"container_image_id": null,
"version": null,
"started": null,
"created": null,
"deployed": "2021-12-11T00:19:24.799615Z",
"configured": null
},
{
"style": "cephadm:v1",
"name": "osd.1",
"fsid": "4cfa6467-6647-41e9-8184-1cacc408265c",
"systemd_unit":
"ceph-4cfa6467-6647-41e9-8184-1cacc408265c(a)osd.1"sd.1",
"enabled": true,
"state": "error",
"container_id": null,
"container_image_name": "ceph/ceph:v16",
"container_image_id": null,
"version": null,
"started": null,
"created": null,
"deployed": "2021-12-11T21:20:02.170515Z",
"configured": null
}
]
Ceph health snippet:
services:
mon: 3 daemons, quorum cephmon02test,cephmon03test,cephmon01test (age 21h)
mgr: cephmon03test(active, since 21h), standbys: cephmon02test
osd: 8 osds: 6 up (since 39m), 8 in
flags noout
Is there a specific way to get those OSDs adopted by cephadm to be shown properly in the
cluster and ceph orchestrator?
I asked the same question elsewhere and was asked if I could see my containers running, I have a reply for that:
Further background info, this cluster was build with 'ceph-deploy' on 12.2.4, I'm not sure if that's an issue _specifically_ for the conversion to cephadm, but I've been able to upgrade from Ubuntu Xenial & Luminous to Ubuntu Focal & Pacific -- it's just this conversion to cephadm that I'm having the issue with. This cluster is _only_ used for RBD devices (via Libvirt).
When I run "bash -x /var/lib/ceph/$FSID/osd.0/unit.run" I find that it's failing after looking for a block device that doesn't exist -- namely /var/lib/ceph/osd/ceph-0. This device was accurate for the ceph-deploy-built OSDs, but after 'cephadm adopt' has been run, the correct block device is '/dev/dm-1' if I'm not mistaken.
Looking at the cephadm logs, it appears this was by design as far as cephadm is concerned, however this is clearly the wrong device and so the containers fail to start.
debug 2021-12-28T03:33:58.368+0000 7f4b3207c080 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (13) Permission denied
debug 2021-12-28T03:33:58.368+0000 7f4b3207c080 1 bluestore(/var/lib/ceph/osd/ceph-0) _mount path /var/lib/ceph/osd/ceph-0
debug 2021-12-28T03:33:58.368+0000 7f4b3207c080 0 bluestore(/var/lib/ceph/osd/ceph-0) _open_db_and_around read-only:0 repair:0
debug 2021-12-28T03:33:58.368+0000 7f4b3207c080 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (13) Permission denied
debug 2021-12-28T03:33:58.368+0000 7f4b3207c080 1 bdev(0x5642f6a9a400 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
debug 2021-12-28T03:33:58.368+0000 7f4b3207c080 -1 bdev(0x5642f6a9a400 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied
debug 2021-12-28T03:33:58.368+0000 7f4b3207c080 -1 osd.0 0 OSD:init: unable to mount object store
debug 2021-12-28T03:33:58.368+0000 7f4b3207c080 -1 ** ERROR: osd init failed: (13) Permission denied
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx<mailto:ceph-users@xxxxxxx>
To unsubscribe send an email to ceph-users-leave@xxxxxxx<mailto:ceph-users-leave@xxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
