1. Stopped osd.60-69: no problem
2. Skipped this and went to #3 to check first
3. Here, `find /etc/systemd/system | grep ceph-volume` returned nothing. I see in that directory
/etc/systemd/system/ceph-disk@60.service # and 61 - 69.
No ceph-volume entries.
On Tue, Nov 6, 2018 at 11:43 AM, Hayashida, Mami <mami.hayashida@xxxxxxx> wrote:
Ok. I will go through this this afternoon and let you guys know the result. Thanks!--On Tue, Nov 6, 2018 at 11:32 AM, Hector Martin <hector@xxxxxxxxxxxxxx> wrote:On 11/7/18 1:00 AM, Hayashida, Mami wrote:
> I see. Thank you for clarifying lots of things along the way -- this
> has been extremely helpful. Neither "df | grep osd" nor "mount | grep
> osd" shows ceph-60 through 69.
OK, that isn't right then. I suggest you try this:
1) bring down OSD 60-69 (systemctl stop ceph-osd@60 etc)
2) move those directories out of the way, as in:
mkdir /var/lib/ceph/osd_old
mv /var/lib/ceph/osd/ceph-6[0-9] /var/lib/ceph/osd_old
(if this all works out you can delete them, just want to make sure you
don't accidentally wipe something important)
2) run `find /etc/systemd/system | grep ceph-volume` and check the
output. You're looking for symlinks in multi-user.target.wants or similar.
There should be a single "ceph-volume@lvm-<id>-<uuid>" entry for each
OSD, and the id and uuid should match the "ceph.osd_id" and
"ceph.osd_fsid" LVM tags from `ceph-volume lvm list`. You can also use
`lvs -o vg_name,name,lv_tags`
If you see anything of the format "ceph-volume@simple-..." then that is
old junk from previous attempts at using ceph-volume. They should be
symlinks and you should delete them and run `systemctl daemon-reload`.
Same story if you see any @lvm symlinks but with incorrect OSD IDs or
fsids. All of this should be recreated by the next step anyway if
deleted, so it should be safe to delete any symlinks in there that you
think might be wrong.
3) Run `ceph-volume lvm activate --all`
At this point `df` and `mount` should show tmpfs mounts for all your LVM
OSDs, and they should be up. List the OSD directories and check that
both `block` and `block.db` entries are symlinks to the right devices.
The right target symlinks should also have been created/enabled in
/etc/systemd/system/multi-user.target.wants.
The LVM dump you provided is correct. I suspect what happened is that
somewhere during this experiment OSDs were activated into the root
filesystem (instead of a tmpfs), perhaps using the ceph-volume simple
mode, perhaps something else. Since all the metadata is in LVM, it's
safe to move or delete all those OSD directories for BlueStore OSDs and
try activating them cleanly again, which hopefully will do the right thing.
In the end this all might fix your device ownership woes too, making the
udev rule unnecessary. If it all works out, try a reboot and see if
everything comes back up as it should.
Mami HayashidaResearch Computing AssociateResearch Computing Infrastructure
University of Kentucky Information Technology Services
301 Rose Street | 102 James F. Hardymon Building
Lexington, KY 40506-0495
mami.hayashida@xxxxxxx
(859)323-7521
Mami Hayashida
Research Computing Associate
Research Computing Infrastructure
University of Kentucky Information Technology Services
301 Rose Street | 102 James F. Hardymon Building
Lexington, KY 40506-0495
mami.hayashida@xxxxxxx
(859)323-7521
University of Kentucky Information Technology Services
301 Rose Street | 102 James F. Hardymon Building
Lexington, KY 40506-0495
mami.hayashida@xxxxxxx
(859)323-7521
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
