I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's
who are end of life. I have some spinners who have their journals on
SSD. Each spinner has a 50GB SSD LVM partition and I want to move those
each to new corresponding partitions.
The new 4TB SSD's I have split into volumes with:
# lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0
# lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0
# lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0
# lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0
# lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1
# lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0
The OSD's that need to have their journals moved are osd.5, osd.4, osd.3
& osd.2. I did osd.5 by deleting the osd and recreating it. However,
that takes a lot of time with a spinner, so I'd rather just move the
Rocksdb and WAL. Of course, this needs to be done on all the nodes.
The osd.4 directory looks like this:
# ls -la /var/lib/ceph/osd/ceph-4/
total 32
drwxrwxrwt 2 ceph ceph 220 Nov 10 14:36 .
drwxr-xr-x 13 ceph ceph 4096 May 9 2024 ..
lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block ->
/dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6
lrwxrwxrwx 1 ceph ceph 39 Nov 10 14:36 block.db ->
/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1
-rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid
-rw------- 1 ceph ceph 37 Nov 10 14:36 fsid
-rw------- 1 ceph ceph 55 Nov 10 14:36 keyring
-rw------- 1 ceph ceph 6 Nov 10 14:36 ready
-rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release
-rw------- 1 ceph ceph 10 Nov 10 14:36 type
-rw------- 1 ceph ceph 2 Nov 10 14:36 whoami
I then take osd4 out of the cluster and stop it. Then I do this:
# ceph-bluestore-tool bluefs-bdev-migrate --path
/var/lib/ceph/osd/ceph-4/ --dev-target
/dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB3 --devs-source
/var/lib/ceph/osd/ceph-4/block.db
inferring bluefs devices from bluestore path
device removed:1 /var/lib/ceph/osd/ceph-4/block.db
device added: 1 /dev/dm-4
# ls -la /var/lib/ceph/osd/ceph-4/
total 32
drwxrwxrwt 2 ceph ceph 220 Nov 11 21:08 .
drwxr-xr-x 13 ceph ceph 4096 May 9 2024 ..
lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block ->
/dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6
lrwxrwxrwx 1 root root 9 Nov 11 21:08 block.db -> /dev/dm-4
-rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid
-rw------- 1 ceph ceph 37 Nov 10 14:36 fsid
-rw------- 1 ceph ceph 55 Nov 10 14:36 keyring
-rw------- 1 ceph ceph 6 Nov 10 14:36 ready
-rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release
-rw------- 1 ceph ceph 10 Nov 10 14:36 type
-rw------- 1 ceph ceph 2 Nov 10 14:36 whoami
Thereafter I do this to load the changes:
# ceph-volume lvm activate --all
--> OSD ID 5 FSID 16173562-980c-42b7-8ea1-941c3a310374 process is
active. Skipping activation
--> OSD ID 40 FSID dc9fcd65-e982-4444-aa8a-29cfc88d27bd process is
active. Skipping activation
--> OSD ID 2 FSID 057da31e-ee92-421f-8ade-619fd052ec1c process is
active. Skipping activation
--> OSD ID 3 FSID c2d820a1-bc92-4a2a-9643-fa8be9acaf6b process is
active. Skipping activation
--> OSD ID 0 FSID 89fd8582-f1af-4c5d-ab52-f2fa0301cc1a process is
active. Skipping activation
--> OSD ID 38 FSID 81234c0b-f257-4e55-951f-f8b8e56f8f3f process is
active. Skipping activation
--> OSD ID 39 FSID 3e3a957b-9405-43f9-9f35-7d45395a6a2a process is
active. Skipping activation
--> Activating OSD ID 4 FSID a3fc7003-564b-421a-a156-71b9ed8506f6
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph
prime-osd-dir --dev
/dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6
--path /var/lib/ceph/osd/ceph-4 --no-mon-config
Running command: /usr/bin/ln -snf
/dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6
/var/lib/ceph/osd/ceph-4/block
Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-21
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4
Running command: /usr/bin/ln -snf
/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 /var/lib/ceph/osd/ceph-4/block.db
Running command: /usr/bin/chown -h ceph:ceph
/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9
Running command: /usr/bin/chown -h ceph:ceph
/var/lib/ceph/osd/ceph-4/block.db
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9
Running command: /usr/bin/systemctl enable
ceph-volume@lvm-4-a3fc7003-564b-421a-a156-71b9ed8506f6
Running command: /usr/bin/systemctl enable --runtime ceph-osd@4
Running command: /usr/bin/systemctl start ceph-osd@4
--> ceph-volume lvm activate successful for osd ID: 4
--> OSD ID 1 FSID ae6a3cea-a613-4436-9940-9e1ab8b74073 process is
active. Skipping activation
--> OSD ID 36 FSID 402d777c-7cc6-4048-945e-a5f470f876ec process is
active. Skipping activation
--> OSD ID 37 FSID 1b97f036-6b3f-4aad-9a60-4b7917ddc0d5 process is
active. Skipping activation
# ls -la /var/lib/ceph/osd/ceph-4/
total 32
drwxrwxrwt 2 ceph ceph 220 Nov 11 21:09 .
drwxr-xr-x 13 ceph ceph 4096 May 9 2024 ..
lrwxrwxrwx 1 ceph ceph 93 Nov 11 21:09 block ->
/dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6
lrwxrwxrwx 1 ceph ceph 39 Nov 11 21:09 block.db ->
/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1
-rw------- 1 ceph ceph 37 Nov 11 21:09 ceph_fsid
-rw------- 1 ceph ceph 37 Nov 11 21:09 fsid
-rw------- 1 ceph ceph 55 Nov 11 21:09 keyring
-rw------- 1 ceph ceph 6 Nov 11 21:09 ready
-rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release
-rw------- 1 ceph ceph 10 Nov 11 21:09 type
-rw------- 1 ceph ceph 2 Nov 11 21:09 whoami
After that the configuration change has reverted to what it was before.
I have tried using the following, but it doesn't help:
# lvchange --deltag
"ceph.db_device=/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB2"
/dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6
# lvchange --addtag
"ceph.db_device=/dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB4"
/dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6
What am I missing to get these changes to be permanent?
regards
Roland
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
