Yes i need specific device paths because all HDD / SSD are the same size / same vendor etc. I group multiple HDDs with an exclusive SSD for cacheing, for example:
spec:
data_devices:
paths:
- /dev/sdh
- /dev/sdi
- /dev/sdj
- /dev/sdk
- /dev/sdl
db_devices:
paths:
- /dev/sdf
filter_logic: AND
objectstore: bluestore
________________________________
Von: Eugen Block <eblock@xxxxxx>
Gesendet: Mittwoch, 2. August 2023 08:13:41
An: ceph-users@xxxxxxx
Betreff: Re: Disk device path changed - cephadm faild to apply osd service
Do you really need device paths in your configuration? You could use
other criteria like disk sizes, vendors, rotational flag etc. If you
really want device paths you'll probably need to ensure they're
persistent across reboots via udev rules.
Zitat von Kilian Ries <mail@xxxxxxxxxxxxxx>:
> Hi,
>
>
> it seems that after reboot / OS update my disk labels / device paths
> may have changed. Since then i get an error like this:
>
>
>
> CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.osd-12-22_hdd-2
>
>
>
> ###
>
>
> RuntimeError: cephadm exited with an error code: 1, stderr:Non-zero
> exit code 1 from /bin/docker run --rm --ipc=host
> --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume
> --privileged --group-add=disk --init -e
> CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:9e2fd45a080aea67d1935d7d9a9025b6db2e8be9173186e068a79a0da5a54ada -e NODE_NAME=ceph-osd07.intern -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=osd-12-22_hdd-2 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/01578d80-6c97-46ba-9327-cb2b13980916:/var/run/ceph:z -v /var/log/ceph/01578d80-6c97-46ba-9327-cb2b13980916:/var/log/ceph:z -v /var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp2cvmr5lf:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpb38cuw7q:/var/lib/ceph/bootstrap-osd/ceph.keyring:z
> quay.io/ceph/ceph@sha256:9e2fd45a080aea67d1935d7d9a9
> 025b6db2e8be9173186e068a79a0da5a54ada lvm batch --no-auto /dev/sdm
> /dev/sdn /dev/sdo /dev/sdp /dev/sdq --db-devices /dev/sdg --yes
> --no-systemd
>
> /bin/docker: stderr Traceback (most recent call last):
>
> /bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module>
>
> /bin/docker: stderr load_entry_point('ceph-volume==1.0.0',
> 'console_scripts', 'ceph-volume')()
>
> /bin/docker: stderr File
> "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in
> __init__
>
> /bin/docker: stderr self.main(self.argv)
>
> /bin/docker: stderr File
> "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line
> 59, in newfunc
>
> /bin/docker: stderr return f(*a, **kw)
>
> /bin/docker: stderr File
> "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in
> main
>
> /bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args)
>
> /bin/docker: stderr File
> "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line
> 194, in dispatch
>
> /bin/docker: stderr instance.main()
>
> /bin/docker: stderr File
> "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py",
> line 46, in main
>
> /bin/docker: stderr terminal.dispatch(self.mapper, self.argv)
>
> /bin/docker: stderr File
> "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line
> 192, in dispatch
>
> /bin/docker: stderr instance = mapper.get(arg)(argv[count:])
>
> /bin/docker: stderr File
> "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py",
> line 348, in __init__
>
> /bin/docker: stderr self.args = parser.parse_args(argv)
>
> /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line
> 1734, in parse_args
>
> /bin/docker: stderr args, argv = self.parse_known_args(args, namespace)
>
> /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line
> 1766, in parse_known_args
>
> /bin/docker: stderr namespace, args =
> self._parse_known_args(args, namespace)
>
> /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line
> 1954, in _parse_known_args
>
> /bin/docker: stderr positionals_end_index =
> consume_positionals(start_index)
>
> /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line
> 1931, in consume_positionals
>
> /bin/docker: stderr take_action(action, args)
>
> /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line
> 1824, in take_action
>
> /bin/docker: stderr argument_values = self._get_values(action,
> argument_strings)
>
> /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line
> 2279, in _get_values
>
> /bin/docker: stderr value = [self._get_value(action, v) for v in
> arg_strings]
>
> /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line
> 2279, in <listcomp>
>
> /bin/docker: stderr value = [self._get_value(action, v) for v in
> arg_strings]
>
> /bin/docker: stderr File "/usr/lib64/python3.6/argparse.py", line
> 2294, in _get_value
>
> /bin/docker: stderr result = type_func(arg_string)
>
> /bin/docker: stderr File
> "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line
> 116, in __call__
>
> /bin/docker: stderr return self._format_device(self._is_valid_device())
>
> /bin/docker: stderr File
> "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line
> 127, in _is_valid_device
>
> /bin/docker: stderr super()._is_valid_device(raise_sys_exit=False)
>
> /bin/docker: stderr File
> "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line
> 104, in _is_valid_device
>
> /bin/docker: stderr super()._is_valid_device()
>
> /bin/docker: stderr File
> "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line
> 69, in _is_valid_device
>
> /bin/docker: stderr super()._is_valid_device()
>
> /bin/docker: stderr File
> "/usr/lib/python3.6/site-packages/ceph_volume/util/arg_validators.py", line
> 47, in _is_valid_device
>
> /bin/docker: stderr raise RuntimeError("Device {} has
> partitions.".format(self.dev_path))
>
> /bin/docker: stderr RuntimeError: Device /dev/sdq has partitions.
>
> Traceback (most recent call last):
>
> File
> "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 9309, in
> <module>
>
> main()
>
> File
> "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 9297, in
> main
>
> r = ctx.func(ctx)
>
> File
> "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1941, in
> _infer_config
>
> return func(ctx)
>
> File
> "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1872, in
> _infer_fsid
>
> return func(ctx)
>
> File
> "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1969, in
> _infer_image
>
> return func(ctx)
>
> File
> "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1859, in
> _validate_fsid
>
> return func(ctx)
>
> File
> "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 5366, in
> command_ceph_volume
>
> out, err, code = call_throws(ctx, c.run_cmd())
>
> File
> "/var/lib/ceph/01578d80-6c97-46ba-9327-cb2b13980916/cephadm.0317efb4d3a353d5a77e82f4a4f52582f06970d6aba66473daecf92e26ee3a51", line 1661, in
> call_throws
>
> raise RuntimeError('Failed command: %s' % ' '.join(command))
>
>
> ###
>
>
>
> /dev/sdg is my boot device at the moment wich was formerly /dev/sda.
>
>
> Is it safe to edit the ceph orch yaml file and change the device
> pathes to the new format? Like this:
>
>
> ceph orch ls --service_name=<service-name> --export > myservice.yaml
>
>
> vi (change device pathes in spec -> data_devices -> path |
> db_devices -> path)
>
>
> ceph orch apply -i myservice.yaml [--dry-run]
>
>
>
> Is that ok / expected behaviour? Or is there a better way?
>
>
> However can see that mgr detects new devices in ceph orch log:
>
>
> mgr.ceph-mon03.lrfomu [INF] Detected new or changed devices on ceph-osd07
>
>
>
> Regards,
>
> Kilian
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
