Hi
We are converting unmanaged OSDs from db/wal on SSD to managed OSDs with
db/wal on NVMe. The boxes had 20 HDDs and 4 SSDs and will be changed to
22 HDDs, 2 SSDs and 2 NVMes, with 11 db/wal partitions on each NVMe for
the HDDs. The old SSDs will be used for a flash pool.
We calculated the block_db_size for the OSDs with db/wal on NVMe as
total bytes / 11 and rounded down, expecting that to fit:
3200631791616 / 11 = 290.966.526.510,5455
Service spec:
"
service_type: osd
service_id: slow
service_name: osd.slow
placement:
hosts:
- doc
- dopey
- happy
- klutzy
- lazy
- sneezy
- smiley
spec:
block_db_size: 290966526510
data_devices:
rotational: 1
db_devices:
rotational: 0
size: '1000G:'
filter_logic: AND
objectstore: bluestore
"
However, the orchestrator/ceph-volume will only fit 10:
"
# lsblk -b /dev/nvme0n1
NAME
MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1
259:0 0 3200631791616 0 disk
├─ceph--20144705--e65d--4143--b917--c0469e54863c-osd--db--298670ae--d218--4af9--8c61--04c93104190c
│
253:21 0 290963062784 0 lvm
├─ceph--20144705--e65d--4143--b917--c0469e54863c-osd--db--9c7f60d0--4757--402a--a66f--3a2e38a3e172
│
253:22 0 290963062784 0 lvm
├─ceph--20144705--e65d--4143--b917--c0469e54863c-osd--db--a30d86bd--dc69--44c9--9a95--893e3c55787f
│
253:23 0 290963062784 0 lvm
├─ceph--20144705--e65d--4143--b917--c0469e54863c-osd--db--7ade6abc--691b--4bb2--a969--5491f4b31eb6
│
253:24 0 290963062784 0 lvm
├─ceph--20144705--e65d--4143--b917--c0469e54863c-osd--db--66325755--6082--421d--9c33--9d13d758709d
│
253:26 0 290963062784 0 lvm
├─ceph--20144705--e65d--4143--b917--c0469e54863c-osd--db--9ca23f86--ceba--4f91--b565--2bcdd4c66352
│
253:28 0 290963062784 0 lvm
├─ceph--20144705--e65d--4143--b917--c0469e54863c-osd--db--1e7b4909--a44b--49e8--9891--31800c0df3ed
│
253:35 0 290963062784 0 lvm
├─ceph--20144705--e65d--4143--b917--c0469e54863c-osd--db--e1701c65--8eec--4311--a274--2bbca17c1ac1
│
253:42 0 290963062784 0 lvm
├─ceph--20144705--e65d--4143--b917--c0469e54863c-osd--db--4af28722--4ba1--400e--aee5--28af3f2f80b5
│
253:45 0 290963062784 0 lvm
└─ceph--20144705--e65d--4143--b917--c0469e54863c-osd--db--51ee229a--edb9--4c2a--8469--0cc006e9f45d
253:47 0 290963062784 0 lvm
"
"
/usr/bin/podman: stderr --> passed data devices: 21 physical, 0 LVM
/usr/bin/podman: stderr --> relative data size: 1.0
/usr/bin/podman: stderr --> passed block_db devices: 2 physical, 0 LVM
/usr/bin/podman: stderr --> 270.98 GB was requested for block_db_size,
but only 270.98 GB can be fulfilled
/usr/bin/podman: stderr time="2022-06-08T07:39:45+02:00" level=warning
msg="Container
a96a3429fdf4487b738f6fb96534ff01697df019dd4893cd07cfc6361ccef26f:
poststop hook 0: executing []: exit status 1"
Traceback (most recent call last):
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.29a5c075eabb1a183db073de0514a72a3722c1b95ce759660d20c1b077d27de0",
line 8826, in <module>
main()
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.29a5c075eabb1a183db073de0514a72a3722c1b95ce759660d20c1b077d27de0",
line 8814, in main
r = ctx.func(ctx)
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.29a5c075eabb1a183db073de0514a72a3722c1b95ce759660d20c1b077d27de0",
line 1889, in _infer_config
return func(ctx)
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.29a5c075eabb1a183db073de0514a72a3722c1b95ce759660d20c1b077d27de0",
line 1830, in _infer_fsid
return func(ctx)
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.29a5c075eabb1a183db073de0514a72a3722c1b95ce759660d20c1b077d27de0",
line 1917, in _infer_image
return func(ctx)
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.29a5c075eabb1a183db073de0514a72a3722c1b95ce759660d20c1b077d27de0",
line 1817, in _validate_fsid
return func(ctx)
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.29a5c075eabb1a183db073de0514a72a3722c1b95ce759660d20c1b077d27de0",
line 5077, in command_ceph_volume
out, err, code = call_throws(ctx, c.run_cmd())
File
"/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/cephadm.29a5c075eabb1a183db073de0514a72a3722c1b95ce759660d20c1b077d27de0",
line 1619, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host
--stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host
--entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init
-e
CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-5-rhel8@sha256:7dc93a9627bf75b2fbfdde6b93d886d41f2f25f2026136e9a93d92de8c8913b9
-e NODE_NAME=sneezy -e CEPH_USE_RANDOM_NONCE=1 -e
CEPH_VOLUME_OSDSPEC_AFFINITY=slow -e CEPH_VOLUME_SKIP_RESTORECON=yes -e
CEPH_VOLUME_DEBUG=1 -v
/var/run/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/run/ceph:z -v
/var/log/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d:/var/log/ceph:z -v
/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/crash:/var/lib/ceph/crash:z
-v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v
/run/lock/lvm:/run/lock/lvm -v
/var/lib/ceph/8ee2d228-ed21-4580-8bbf-0649f229e21d/selinux:/sys/fs/selinux:ro
-v /:/rootfs -v /tmp/ceph-tmp8q_3l7o0:/etc/ceph/ceph.conf:z -v
/tmp/ceph-tmp37fkpwdx:/var/lib/ceph/bootstrap-osd/ceph.keyring:z
registry.redhat.io/rhceph/rhceph-5-rhel8@sha256:7dc93a9627bf75b2fbfdde6b93d886d41f2f25f2026136e9a93d92de8c8913b9
lvm batch --no-auto /dev/sdaa /dev/sdd /dev/sde /dev/sdf /dev/sdg
/dev/sdh /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp
/dev/sdq /dev/sdr /dev/sds /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy
--db-devices /dev/nvme0n1 /dev/nvme1n1 --block-db-size 290966526510
--yes --no-systemd
"
Our block_db_size size is 290966526510 but as seen on lsblk it's rounded
down to 290963062784 to avoid the fraction:
290966526510 / 4.194.304 = 69.371,82581663132 extents pr db
290963062784 / 4.194.304 = 69.371 extents pr db
69.371 x 11 = 763.081 total extents
69.371 x 10 = 693.710 used extents
"
pvdisplay /dev/nvme0n1
PE Size 4.00 MiB
Total PE 763089
Free PE 69379
Allocated PE 693710
"
There's 69.379 extents free and we need 69.371 for the last db/wal so it
should fit? What did we miss?
Mvh.
Torkil
--
Torkil Svensgaard
Systems Administrator
Danish Research Centre for Magnetic Resonance DRCMR, Section 714
Copenhagen University Hospital Amager and Hvidovre
Kettegaard Allé 30, 2650 Hvidovre, Denmark
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
