Hi Frank,
I agree there's a problem there. Howewever, to clarify: the json file
already contains the /dev/sdq1 path (at data:path) and the "simple activate"
is just reading the file. I.e. the problem lies with the json file creator,
which was the "ceph-volume simple scan" step.
For fix your immediate issue I'd suggest fixing the existing json files to
point data:path to the by-partuuid path. That should allow your "simple
activate" to work with the stable paths.
In general it seems something in the "ceph-volume simple scan" is doing a
"realpath()" or "readlink()" on the by-partuuid path to get to the /dev/sdq1
path.
Oh, it's likely this is the culprit:
src/ceph-volume/ceph_volume/devices/simple/scan.py
class Scan(object):
...
def scan_device(self, path):
...
if os.path.islink(path):
device = os.readlink(path)
else:
device = path
I'm not sure what the general fix might be - there may be good reason to
prefer the symlink destination path, e.g. the next steps use the 'device'
var to look for lvm info which may not work with the original symlink path.
I'll leave it up to the developers to work out a proper solution to this!
If you haven't already it's probably worth opening a ticket.
Cheers,
Chris
On Thu, Mar 11, 2021 at 02:19:57PM +0000, Frank Schilder wrote:
Hi Chris,
I found the problem. "ceph-volume simple activate" modifies the OSD's meta data in an invalid way.
On a pre lvm-converted ceph-disk OSD I had in my cupboard:
[root@ceph-adm:ceph-20 ~]# mount /dev/sdq1 mnt
[root@ceph-adm:ceph-20 ~]# ls -l mnt
[...]
lrwxrwxrwx. 1 ceph ceph 58 Mar 15 2019 block -> /dev/disk/by-partuuid/a1e5ef7d-9bab-4911-abe5-9075b91d88a4
[..]
[root@ceph-adm:ceph-20 ~]# umount mnt
[root@ceph-adm:ceph-20 ~]# cat /etc/ceph/osd/59-9b88d6ec-87a4-4640-b80e-81d3d56fac15.json
{
"active": "ok",
"block": {
"path": "/dev/disk/by-partuuid/a1e5ef7d-9bab-4911-abe5-9075b91d88a4",
"uuid": "a1e5ef7d-9bab-4911-abe5-9075b91d88a4"
},
"block_uuid": "a1e5ef7d-9bab-4911-abe5-9075b91d88a4",
"bluefs": 1,
"ceph_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9",
"cluster_name": "ceph",
"data": {
"path": "/dev/sdq1",
"uuid": "9b88d6ec-87a4-4640-b80e-81d3d56fac15"
},
"fsid": "9b88d6ec-87a4-4640-b80e-81d3d56fac15",
"keyring": "AQBP4opcBeCYOxAA4sOpTthNE6T28WUf4Bgm3w==",
"kv_backend": "rocksdb",
"magic": "ceph osd volume v026",
"mkfs_done": "yes",
"none": "",
"ready": "ready",
"require_osd_release": "",
"type": "bluestore",
"whoami": 59
}
Now, "ceph-volume simple activate" modifies the symlink "block" to point to an unstable path:
[root@ceph-adm:ceph-20 ~]# ceph-volume simple activate --file "/etc/ceph/osd/59-9b88d6ec-87a4-4640-b80e-81d3d56fac15.json" --no-systemd
Running command: /usr/bin/mount -v /dev/sdq1 /var/lib/ceph/osd/ceph-59
stdout: mount: /dev/sdq1 mounted on /var/lib/ceph/osd/ceph-59.
Running command: /usr/bin/ln -snf /dev/sdq2 /var/lib/ceph/osd/ceph-59/block
Running command: /usr/bin/chown -R ceph:ceph /dev/sdq2
--> Skipping enabling of `simple` systemd unit
--> Skipping masking of ceph-disk systemd units
--> Skipping enabling and starting OSD simple systemd unit because --no-systemd was used
--> Successfully activated OSD 59 with FSID 9b88d6ec-87a4-4640-b80e-81d3d56fac15
Its the command "/usr/bin/ln -snf /dev/sdq2 /var/lib/ceph/osd/ceph-59/block" that destroys the integrity of the OSD. If you reboot the machine and the devices get different names, the next execution of "ceph-volume simple scan" will produce a corrupted meta data file. This will also happen if you move a converted OSD to another host and try to scan+start it.
The change of the symbolic link to an unstable device path is a critical bug and I don't even understand why it happens in the first place. There is no point and the only valid link target would be "/dev/disk/by-partuuid/a1e5ef7d-9bab-4911-abe5-9075b91d88a4" any ways.
I can work aroud that by resetting the link to its correct value after activation. However, this should really be fixed.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
