On 19/03/2025 11:31, Torkil Svensgaard wrote:
On 19/03/2025 08:33, Torkil Svensgaard wrote:
Hi
I am adding HDDs to a replacement server which will fit 34 HDDs and 2
SATA SSDs, and has 4 NVMe devices for DB/WAL.
The orchaetrator now fails to create any more OSDs due to:
"
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size,
but only 248.40 GB can be fulfilled
"
So I kept adding more HDDs since that has to be done anyway and noticed
that the error changed:
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size,
but only 248.40 GB can be fulfilled
Added 4 drives
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size,
but only 212.92 GB can be fulfilled
Added 4 drives
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size,
but only 186.30 GB can be fulfilled
So it looks like ceph-volume is doing some sort of total calculation
with the drives and devices passed and thinks there is too little room
for some reason.
Workaround seems to be adding the drives one at a time.
Or not, that only worked for a few drives, until every NVMe had 5 db
partitions. We can't get the math to fit exactly, but we have this:
"
nvme2n1
259:2 0 2.9T 0 disk
├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db
│
253:7 0 270.1G 0 lvm
├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--feb3bd0f--858b--45c5--a1ba--c0c77f34dc0d
│
253:15 0 270.1G 0 lvm
├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf
│
253:21 0 270.1G 0 lvm
├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--d4f3c31f--2327--4662--9517--f86dbe35c510
│
253:27 0 270.1G 0 lvm
└─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--eebae904--4ae7--4550--8c3a--0af3ef1fec1c
253:29 0 270.1G 0 lvm
nvme0n1
259:3 0 2.9T 0 disk
├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e--a197--4ecf--a021--94b135039661
│
253:3 0 270.1G 0 lvm
├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348--fc04--4e89--823d--5ebdf26e0408
│
253:19 0 270.1G 0 lvm
├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--ee274757--51d8--48e3--a41e--b2b321da7170
│
253:25 0 270.1G 0 lvm
├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--79ce0245--338a--432c--92dd--1437dcaf3917
│
253:35 0 270.1G 0 lvm
└─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--e4e2b277--03c3--47aa--bf32--12f63faee4e5
253:37 0 270.1G 0 lvm
nvme3n1
259:6 0 1.5T 0 disk
├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1--e23c--4390--a034--54d4a224b2a2
│
253:5 0 270.1G 0 lvm
├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5--ad6d--453d--91bc--8a52f1bfa533
│
253:13 0 270.1G 0 lvm
├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--d34e427d--749f--460f--bc15--db5ab3900a8e
│
253:39 0 270.1G 0 lvm
├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--2342ed1b--8eec--4fc6--9176--bf5a149d3c30
│
253:41 0 270.1G 0 lvm
└─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--e305cdcf--4a1a--435e--9d61--65c672b3ca5b
253:43 0 270.1G 0 lvm
nvme1n1
259:7 0 1.5T 0 disk
├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--36f82df5--953b--450a--b9d7--5e2ba334a0e7
│
253:9 0 270.1G 0 lvm
├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827--ff63--43d4--a4e8--43681ad8229b
│
253:17 0 270.1G 0 lvm
├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498--d3b9--4ddd--b752--ab95e86bc027
│
253:23 0 270.1G 0 lvm
├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a
│
253:31 0 270.1G 0 lvm
└─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0
253:33 0 270.1G 0 lvm
"
The two 1.5T NVMe do not have room for any more 270G db partitions, but
the two 2.9T ones have plenty of room.
The error is similar to what I had originally, so I think ceph-volume is
simply trying to use one of the small NVMes and not the bigger ones with
free space.
"
2025-03-19T11:19:00.031007+0000 mgr.ceph-flash1.erhakb [ERR] Failed to
apply osd.slow spec
DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd
service_id: slow
service_name: osd.slow
placement:
host_pattern: '*'
spec:
block_db_size: 290000000000
data_devices:
rotational: 1
db_devices:
rotational: 0
size: 1000G:7000G
filter_logic: AND
objectstore: bluestore
''')): cephadm exited with an error code: 1, stderr:Inferring config
/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf
Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host
--stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host
--entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init
-e
CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de
-e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e
CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v
/var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v
/var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v
/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z
-v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v
/run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v
/run/lock/lvm:/run/lock/lvm -v
/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro
-v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v
/tmp/ceph-tmpc2zygv3o:/etc/ceph/ceph.conf:z -v
/tmp/ceph-tmpk9buf727:/var/lib/ceph/bootstrap-osd/ceph.keyring:z
quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de
lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg
/dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq
/dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy
--db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
--block-db-size 290000000000 --yes --no-systemd
/usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM
/usr/bin/podman: stderr --> relative data size: 1.0
/usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size,
but only 248.40 GB can be fulfilled
Traceback (most recent call last):
File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py",
line 5581, in <module>
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py",
line 5569, in main
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py",
line 409, in _infer_config
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py",
line 324, in _infer_fsid
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py",
line 437, in _infer_image
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py",
line 311, in _validate_fsid
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py",
line 3314, in command_ceph_volume
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py",
line 310, in call_throws
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host
--stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host
--entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init
-e
CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de
-e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e
CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v
/var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v
/var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v
/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z
-v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v
/run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v
/run/lock/lvm:/run/lock/lvm -v
/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro
-v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v
/tmp/ceph-tmpc2zygv3o:/etc/ceph/ceph.conf:z -v
/tmp/ceph-tmpk9buf727:/var/lib/ceph/bootstrap-osd/ceph.keyring:z
quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de
lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg
/dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq
/dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy
--db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1
--block-db-size 290000000000 --yes --no-systemd
"
Mvh.
Torkil
Mvh.
Torkil
It looks to me like 1 of the 4 NVMe devices is indeed too full to fit
another DB/WAL partition but ceph-volume is being passed all 4 devices
so it should just pick another? Also, it has been creating DB/WAL
partitions across all 4 devices up til now, so it's not like it only
looks at the first device passed.
Suggestions?
"
2025-03-19T07:20:11.898279+0000 mgr.ceph-flash1.erhakb [INF] Detected
new or changed devices on franky
2025-03-19T07:21:31.960784+0000 mgr.ceph-flash1.erhakb [ERR] Failed to
apply osd.slow spec
DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd
service_id: slow
service_name: osd.slow
placement:
host_pattern: '*'
spec:
block_db_size: 290000000000
data_devices:
rotational: 1
db_devices:
rotational: 0
size: 1000G:7000G
filter_logic: AND
objectstore: bluestore
''')): cephadm exited with an error code: 1, stderr:Inferring config /
var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/ceph.conf
Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-
signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --
entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init
-e CONTAINER_IMAGE=quay.io/ceph/
ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd
/usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM
/usr/bin/podman: stderr --> relative data size: 1.0
/usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size,
but only 248.40 GB can be fulfilled
Traceback (most recent call last):
File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module>
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --
stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host
-- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --
init -e CONTAINER_IMAGE=quay.io/ceph/
ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/serve.py", line 602, in
_apply_all_services
if self._apply_service(spec):
File "/usr/share/ceph/mgr/cephadm/serve.py", line 721, in
_apply_service
self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec))
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 79, in
create_from_spec
ret = self.mgr.wait_async(all_hosts())
File "/usr/share/ceph/mgr/cephadm/module.py", line 815, in wait_async
return self.event_loop.get_result(coro, timeout)
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result
return future.result(timeout)
File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in
result
return self.__get_result()
File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in
__get_result
raise self._exception
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 76, in
all_hosts
return await gather(*futures)
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 63, in
create_from_spec_one
ret_msg = await self.create_single_host(
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 95, in
create_single_host
raise RuntimeError(
RuntimeError: cephadm exited with an error code: 1, stderr:Inferring
config /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/config/
ceph.conf
Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-
signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --
entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init
-e CONTAINER_IMAGE=quay.io/ceph/
ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd
/usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM
/usr/bin/podman: stderr --> relative data size: 1.0
/usr/bin/podman: stderr --> passed block_db devices: 4 physical, 0 LVM
/usr/bin/podman: stderr --> 270.08 GB was requested for block_db_size,
but only 248.40 GB can be fulfilled
Traceback (most recent call last):
File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5581, in <module>
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 5569, in main
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 409, in _infer_config
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 324, in _infer_fsid
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 437, in _infer_image
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 311, in _validate_fsid
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/__main__.py", line 3314, in command_ceph_volume
File "/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/
cephadm.c6d8d2eb72a60267f2844dd3167700619f1207413db7701f1827abf652e86a11/cephadmlib/call_wrappers.py", line 310, in call_throws
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --
stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host
-- entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --
init -e CONTAINER_IMAGE=quay.io/ceph/
ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de -e NODE_NAME=franky -e CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v /var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpt7wifcjq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmphx5nhdsi:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:41d3f5e46ff7de28544cc8869fdea13fca824dcef83936cb3288ed9de935e4de lvm batch --no-auto /dev/sda /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 290000000000 --yes --no-systemd
"
"
[root@franky ~]# lsblk
NAME MAJ:MIN RM
SIZE RO TYPE MOUNTPOINTS
sda 8:0 0
2.7T 0 disk
└─ceph--6c2f1506--e2b6--4c4a--a808--e6b74363010f-osd--
block--76e673e9-- d0c0--46bb--8c3c--e666ac5cef3b
253:2 0
2.7T 0 lvm
sdb 8:16 0
223.6G 0 disk
├─sdb1 8:17 0
1000M 0 part
│ └─md0 9:0 0
999M 0 raid1 /boot
├─sdb2 8:18 0
512M 0 part
│ └─md126 9:126 0
511.9M 0 raid1 /boot/efi
└─sdb3 8:19 0
222.1G 0 part
└─md127 9:127 0
222G 0 raid1
├─rhel-root 253:0 0
221G 0 lvm /var/lib/containers/storage/overlay
│ /
└─rhel-swap 253:1 0
1G 0 lvm
sdc 8:32 0
223.6G 0 disk
├─sdc1 8:33 0
1000M 0 part
│ └─md0 9:0 0
999M 0 raid1 /boot
├─sdc2 8:34 0
512M 0 part
│ └─md126 9:126 0
511.9M 0 raid1 /boot/efi
└─sdc3 8:35 0
222.1G 0 part
└─md127 9:127 0
222G 0 raid1
├─rhel-root 253:0 0
221G 0 lvm /var/lib/containers/storage/overlay
│ /
└─rhel-swap 253:1 0
1G 0 lvm
sdd 8:48 0
1.8T 0 disk
└─ceph--9b031aa3--1d29--4709--9870--6ac3b48abf74-osd--block--
b7330837-- b986--46d7--9e28--57db65945098
253:4 0
1.8T 0 lvm
sde 8:64 0
2.7T 0 disk
└─ceph--d400720f--8236--4689--b3ac--0300514ac42c-osd--
block--0575ecbc--4acb--4cb1--a9a7--607d63a891b3
253:6 0
2.7T 0 lvm
sdf 8:80 0
1.8T 0 disk
└─ceph--a4a3f8ea--6c2e--4f2d--ac57--fa8e8cfb02b0-osd--
block--8a0c4a74-- fedc--46b3--b2a8--d60fd18a37c1
253:8 0
1.8T 0 lvm
sdg 8:96 0
2.7T 0 disk
└─ceph--3a664362--832a--4419--99ae--595a2bb86749-osd--block--
c96afbde--9c71--408f--a961--68c6d14a701f
253:12 0
2.7T 0 lvm
sdh 8:112 0
16.4T 0 disk
└─ceph--6fa9be6b--485b--4433--8e05--a17a6a9d0b70-osd--
block--29747e0e--9c71--44e6--b750--93a7878977ee
253:14 0
16.4T 0 lvm
sdi 8:128 0
16.4T 0 disk
└─ceph--9d1359c4--4af6--489e--974a--c89a5b2160aa-osd--
block--618f7582-- fce0--41f6--aad8--6d0231ef303a
253:16 0
16.4T 0 lvm
sdj 8:144 0
1.8T 0 disk
└─ceph--5a61c09b--027e--4882--8b93--6688d9e98dfa-osd--
block--8e9b21b6-- cc39--4c7b--b5f8--9e83e33fa146
253:18 0
1.8T 0 lvm
sdk 8:160 0
447.1G 0 disk
└─ceph--4b4f3bd9--16be--493a--8e35--84643d1b327c-osd--
block--14f68253-- b370--4300--a319--0c39311a34e1
253:10 0
447.1G 0 lvm
sdl 8:176 0
186.3G 0 disk
└─ceph--fc7e9d84--c650--4a4b--9b53--6a748c9dcad8-osd--
block--2b475461-- a85d--4e7b--a7e2--8ab1c9d14c6e
253:11 0
186.3G 0 lvm
sdm 8:192 0
3.6T 0 disk
└─ceph--ec427ec1--e621--4981--9a58--d9cdf7a909b5-osd--block--
f08c7e71-- ddf5--4939--8f0c--42396de2210b
253:26 0
3.6T 0 lvm
sdn 8:208 0
2.7T 0 disk
└─ceph--9ee3a783--1aa1--4520--83b7--d804972bc7b2-osd--
block--40cd9045--587a--4831--95f7--607c019ef862
253:20 0
2.7T 0 lvm
sdo 8:224 0
2.7T 0 disk
└─ceph--7807748d--305c--4cdf--9812--0a6005e99579-osd--block--
c2087805--441c--422a--b12f--de10a75b7e0b
253:22 0
2.7T 0 lvm
sdp 8:240 0
3.6T 0 disk
└─ceph--d67be0c9--859f--4ac5--8895--18a50fa2a2d7-osd--block--
c0ef5f16--4bcd--4390--abf8--260c5913cb14
253:24 0
3.6T 0 lvm
sdq 65:0 0
3.6T 0 disk
└─ceph--f8d07270--2bcd--49bc--bd43--ee3f2fbaa5ff-osd--
block--8f783ade--6da6--4967--b87c--fb6d1827460f
253:28 0
3.6T 0 lvm
sdr 65:16 0
16.4T 0 disk
└─ceph--71ff9863--a1e1--4ca3--ad23--732b207d4ee4-osd--block--
a3dbca80--9cf5--4c98--ad9b--b38803230b1f
253:30 0
16.4T 0 lvm
sds 65:32 0
3.6T 0 disk
└─ceph--172e2cab--1835--4b7b--a765--3530092e99dd-osd--
block--032fc777--69d9--4f39--8fe6--485e0959ce66
253:32 0
3.6T 0 lvm
sdt 65:48 0
16.4T 0 disk
└─ceph--8bad9f81--5851--464d--89bc--fa645e05934e-osd--block--
ff12d109-- dfaa--4608--a5e2--36ec39623f36
253:34 0
16.4T 0 lvm
sdu 65:64 0
3.6T 0 disk
└─ceph--1a629cf6--c438--4d97--b90d--7b56032d10d5-osd--block--
b3e38c3d-- e3c0--49c2--9e96--93f9e4c909d4
253:36 0
3.6T 0 lvm
sdv 65:80 0
3.6T 0 disk
sdw 65:96 0
5.5T 0 disk
sdx 65:112 0
3.6T 0 disk
sdy 65:128 0
16.4T 0 disk
nvme2n1 259:2 0
2.9T 0 disk
├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--
d28b3fa5--5ace--49a1--ad3d--fc4d14f1b8db
│ 253:7 0
270.1G 0 lvm
├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--
feb3bd0f--858b--45c5--a1ba--c0c77f34dc0d
│ 253:15 0
270.1G 0 lvm
├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--
db--1ddcc845--799a--4d0d--96f1--90078e2cf0cf
│ 253:21 0
270.1G 0 lvm
├─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--
d4f3c31f--2327--4662--9517--f86dbe35c510
│ 253:27 0
270.1G 0 lvm
└─ceph--9fc0a64d--9ab8--4b12--9a8f--6a48e6c95211-osd--db--
eebae904--4ae7--4550--8c3a--0af3ef1fec1c
253:29 0
270.1G 0 lvm
nvme0n1 259:3 0
2.9T 0 disk
├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--12f47f4e--
a197--4ecf--a021--94b135039661
│ 253:3 0
270.1G 0 lvm
├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--da108348--
fc04--4e89--823d--5ebdf26e0408
│ 253:19 0
270.1G 0 lvm
├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--
ee274757--51d8--48e3--a41e--b2b321da7170
│ 253:25 0
270.1G 0 lvm
├─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--
db--79ce0245--338a--432c--92dd--1437dcaf3917
│ 253:35 0
270.1G 0 lvm
└─ceph--c2aeb797--fc4a--4054--9d12--7c5550ac1641-osd--db--
e4e2b277--03c3--47aa--bf32--12f63faee4e5
253:37 0
270.1G 0 lvm
nvme3n1 259:6 0
1.5T 0 disk
├─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--3f4c63c1--
e23c--4390--a034--54d4a224b2a2
│ 253:5 0
270.1G 0 lvm
└─ceph--20c0fa86--7668--4139--bf39--f7bb8b2e1623-osd--db--a45b02f5--
ad6d--453d--91bc--8a52f1bfa533
253:13 0
270.1G 0 lvm
nvme1n1 259:7 0
1.5T 0 disk
├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--
db--36f82df5--953b--450a--b9d7--5e2ba334a0e7
│ 253:9 0
270.1G 0 lvm
├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--857ec827--
ff63--43d4--a4e8--43681ad8229b
│ 253:17 0
270.1G 0 lvm
├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--db--b370a498--
d3b9--4ddd--b752--ab95e86bc027
│ 253:23 0
270.1G 0 lvm
├─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--
db--5468cc56--54aa--4965--8a32--cf4d6b29fb3a
│ 253:31 0
270.1G 0 lvm
└─ceph--f1df0c25--920d--4d13--ba48--ad56302a4099-osd--
db--6d4941f5--96d6--4e67--81f5--77780e1a3ab0
253:33 0
270.1G 0 lvm
[root@franky ~]#
"
Mvh.
Torkil
--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: torkil@xxxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
