Hi Roland, On Mon, Nov 11, 2024, 20:16 Roland Giesler <roland@xxxxxxxxxxxxxx> wrote: > I have ceph 17.2.6 on a proxmox cluster and want to replace some ssd's > who are end of life. I have some spinners who have their journals on > SSD. Each spinner has a 50GB SSD LVM partition and I want to move those > each to new corresponding partitions. > > The new 4TB SSD's I have split into volumes with: > > # lvcreate -n NodeA-nvme-LV-RocksDB1 -L 47.69g NodeA-nvme0 > # lvcreate -n NodeA-nvme-LV-RocksDB2 -L 47.69g NodeA-nvme0 > # lvcreate -n NodeA-nvme-LV-RocksDB3 -L 47.69g NodeA-nvme0 > # lvcreate -n NodeA-nvme-LV-RocksDB4 -L 47.69g NodeA-nvme0 > # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme1 > # lvcreate -n NodeA-nvme-LV-data -l 100%FREE NodeA-nvme0 > I caution the mix of DB/WAL partitions with other applications. The performance profile may not be suited for shared use. And depending on the use case the ~48GB might not be big enough to hinder DB spillover. See the current size when querying the OSD. The OSD's that need to have their journals moved are osd.5, osd.4, osd.3 > & osd.2. I did osd.5 by deleting the osd and recreating it. However, > that takes a lot of time with a spinner, so I'd rather just move the > Rocksdb and WAL. Of course, this needs to be done on all the nodes. > The osd.4 directory looks like this: > > # ls -la /var/lib/ceph/osd/ceph-4/ > total 32 > drwxrwxrwt 2 ceph ceph 220 Nov 10 14:36 . > drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. > lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> > > /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 > lrwxrwxrwx 1 ceph ceph 39 Nov 10 14:36 block.db -> > /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 > -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid > -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid > -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring > -rw------- 1 ceph ceph 6 Nov 10 14:36 ready > -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release > -rw------- 1 ceph ceph 10 Nov 10 14:36 type > -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami > > I then take osd4 out of the cluster and stop it. Then I do this: > > # ceph-bluestore-tool bluefs-bdev-migrate --path > /var/lib/ceph/osd/ceph-4/ --dev-target > /dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB3 --devs-source > /var/lib/ceph/osd/ceph-4/block.db > inferring bluefs devices from bluestore path > device removed:1 /var/lib/ceph/osd/ceph-4/block.db > device added: 1 /dev/dm-4 > # ls -la /var/lib/ceph/osd/ceph-4/ > total 32 > drwxrwxrwt 2 ceph ceph 220 Nov 11 21:08 . > drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. > lrwxrwxrwx 1 ceph ceph 93 Nov 10 14:36 block -> > > /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 > lrwxrwxrwx 1 root root 9 Nov 11 21:08 block.db -> /dev/dm-4 > -rw------- 1 ceph ceph 37 Nov 10 14:36 ceph_fsid > -rw------- 1 ceph ceph 37 Nov 10 14:36 fsid > -rw------- 1 ceph ceph 55 Nov 10 14:36 keyring > -rw------- 1 ceph ceph 6 Nov 10 14:36 ready > -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release > -rw------- 1 ceph ceph 10 Nov 10 14:36 type > -rw------- 1 ceph ceph 2 Nov 10 14:36 whoami > > Thereafter I do this to load the changes: > > # ceph-volume lvm activate --all > --> OSD ID 5 FSID 16173562-980c-42b7-8ea1-941c3a310374 process is > active. Skipping activation > --> OSD ID 40 FSID dc9fcd65-e982-4444-aa8a-29cfc88d27bd process is > active. Skipping activation > --> OSD ID 2 FSID 057da31e-ee92-421f-8ade-619fd052ec1c process is > active. Skipping activation > --> OSD ID 3 FSID c2d820a1-bc92-4a2a-9643-fa8be9acaf6b process is > active. Skipping activation > --> OSD ID 0 FSID 89fd8582-f1af-4c5d-ab52-f2fa0301cc1a process is > active. Skipping activation > --> OSD ID 38 FSID 81234c0b-f257-4e55-951f-f8b8e56f8f3f process is > active. Skipping activation > --> OSD ID 39 FSID 3e3a957b-9405-43f9-9f35-7d45395a6a2a process is > active. Skipping activation > --> Activating OSD ID 4 FSID a3fc7003-564b-421a-a156-71b9ed8506f6 > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 > Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph > prime-osd-dir --dev > /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 > > --path /var/lib/ceph/osd/ceph-4 --no-mon-config > Running command: /usr/bin/ln -snf > /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 > > /var/lib/ceph/osd/ceph-4/block > Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-4/block > Running command: /usr/bin/chown -R ceph:ceph /dev/dm-21 > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 > Running command: /usr/bin/ln -snf > /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 /var/lib/ceph/osd/ceph-4/block.db > Running command: /usr/bin/chown -h ceph:ceph > /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 > Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 > Running command: /usr/bin/chown -h ceph:ceph > /var/lib/ceph/osd/ceph-4/block.db > Running command: /usr/bin/chown -R ceph:ceph /dev/dm-9 > Running command: /usr/bin/systemctl enable > ceph-volume@lvm-4-a3fc7003-564b-421a-a156-71b9ed8506f6 > Running command: /usr/bin/systemctl enable --runtime ceph-osd@4 > Running command: /usr/bin/systemctl start ceph-osd@4 > --> ceph-volume lvm activate successful for osd ID: 4 > --> OSD ID 1 FSID ae6a3cea-a613-4436-9940-9e1ab8b74073 process is > active. Skipping activation > --> OSD ID 36 FSID 402d777c-7cc6-4048-945e-a5f470f876ec process is > active. Skipping activation > --> OSD ID 37 FSID 1b97f036-6b3f-4aad-9a60-4b7917ddc0d5 process is > active. Skipping activation > > # ls -la /var/lib/ceph/osd/ceph-4/ > total 32 > drwxrwxrwt 2 ceph ceph 220 Nov 11 21:09 . > drwxr-xr-x 13 ceph ceph 4096 May 9 2024 .. > lrwxrwxrwx 1 ceph ceph 93 Nov 11 21:09 block -> > > /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 > lrwxrwxrwx 1 ceph ceph 39 Nov 11 21:09 block.db -> > /dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB1 > -rw------- 1 ceph ceph 37 Nov 11 21:09 ceph_fsid > -rw------- 1 ceph ceph 37 Nov 11 21:09 fsid > -rw------- 1 ceph ceph 55 Nov 11 21:09 keyring > -rw------- 1 ceph ceph 6 Nov 11 21:09 ready > -rw------- 1 ceph ceph 3 Nov 10 14:36 require_osd_release > -rw------- 1 ceph ceph 10 Nov 11 21:09 type > -rw------- 1 ceph ceph 2 Nov 11 21:09 whoami > > After that the configuration change has reverted to what it was before. > > I have tried using the following, but it doesn't help: > # lvchange --deltag > "ceph.db_device=/dev/NodeA-nvme2/NodeA-nvme-LV-RocksDB2" > > /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 > # lvchange --addtag > "ceph.db_device=/dev/NodeA-nvme0/NodeA-nvme-LV-RocksDB4" > > /dev/ceph-4ef87fd1-040b-4914-8290-4ce43f4e4412/osd-block-a3fc7003-564b-421a-a156-71b9ed8506f6 > > What am I missing to get these changes to be permanent? > Likely just an issue with the order of execution. But there is an easier way to do the move. See: https://docs.ceph.com/en/quincy/ceph-volume/lvm/migrate/ Cheers, Alwin -- > Alwin Antreich Head of Training and Proxmox Services croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges, Andy Muthmann - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io/ _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |