Hi Igor,
what's your Ceph version?
12.2.8 (SES 5.5 - patched to the latest version)
Can you also check the output for ceph-bluestore-tool show-label -p <path to osd>
ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0/
infering bluefs devices from bluestore path
{
"/var/lib/ceph/osd/ceph-0//block": {
"osd_uuid": "1e5b3908-20b1-41e4-b6eb-f5636d20450b",
"size": 8001457295360,
"btime": "2018-06-29 23:43:12.088842",
"description": "main",
"bluefs": "1",
"ceph_fsid": "a2222146-6561-307e-b032-c5cee2ee520c",
"kv_backend": "rocksdb",
"magic": "ceph osd volume v026",
"mkfs_done": "yes",
"ready": "ready",
"whoami": "0"
},
"/var/lib/ceph/osd/ceph-0//block.wal": {
"osd_uuid": "1e5b3908-20b1-41e4-b6eb-f5636d20450b",
"size": 524288000,
"btime": "2018-06-29 23:43:12.098690",
"description": "bluefs wal"
},
"/var/lib/ceph/osd/ceph-0//block.db": {
"osd_uuid": "1e5b3908-20b1-41e4-b6eb-f5636d20450b",
"size": 524288000,
"btime": "2018-06-29 23:43:12.098023",
"description": "bluefs db"
}
}
It should report 'size' labels for every volume, please check they contain new values.
That's exactly the problem, whether "ceph-bluestore-tool show-label" nor "ceph daemon osd.0 perf dump|jq '.bluefs'" did recognize the new sizes. But we are 100% sure the new devices are used as we already deleted the old once...
We tried to delete the "key" "size" to add one with the new value but:ceph-bluestore-tool rm-label-key --dev /var/lib/ceph/osd/ceph-0/block.db -k size
key 'size' not present
even if:
ceph-bluestore-tool show-label --dev /var/lib/ceph/osd/ceph-0/block.db
{
"/var/lib/ceph/osd/ceph-0/block.db": {
"osd_uuid": "1e5b3908-20b1-41e4-b6eb-f5636d20450b",
"size": 524288000,
"btime": "2018-06-29 23:43:12.098023",
"description": "bluefs db"
}
}
So it looks like the key "size" is "read-only"?
Thanks, Igor On 11/20/2018 5:29 PM, Florian Engelmann wrote:Hi,today we migrated all of our rocksdb and wal devices to new once. The new once are much bigger (500MB for wal/db -> 60GB db and 2G WAL) and LVM based.We migrated like: export OSD=x systemctl stop ceph-osd@$OSD lvcreate -n db-osd$OSD -L60g data || exit 1 lvcreate -n wal-osd$OSD -L2g data || exit 1dd if=/var/lib/ceph/osd/ceph-$OSD/block.wal of=/dev/data/wal-osd$OSD bs=1M || exit 1 dd if=/var/lib/ceph/osd/ceph-$OSD/block.db of=/dev/data/db-osd$OSD bs=1M || exit 1rm -v /var/lib/ceph/osd/ceph-$OSD/block.db || exit 1 rm -v /var/lib/ceph/osd/ceph-$OSD/block.wal || exit 1ln -vs /dev/data/db-osd$OSD /var/lib/ceph/osd/ceph-$OSD/block.db || exit 1 ln -vs /dev/data/wal-osd$OSD /var/lib/ceph/osd/ceph-$OSD/block.wal || exit 1chown -c ceph:ceph $(realpath /dev/data/db-osd$OSD) || exit 1 chown -c ceph:ceph $(realpath /dev/data/wal-osd$OSD) || exit 1 chown -ch ceph:ceph /var/lib/ceph/osd/ceph-$OSD/block.db || exit 1 chown -ch ceph:ceph /var/lib/ceph/osd/ceph-$OSD/block.wal || exit 1ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-$OSD/ || exit 1systemctl start ceph-osd@$OSDEverything went fine but it looks like the db and wal size is still the old one:ceph daemon osd.0 perf dump|jq '.bluefs' { "gift_bytes": 0, "reclaim_bytes": 0, "db_total_bytes": 524279808, "db_used_bytes": 330301440, "wal_total_bytes": 524283904, "wal_used_bytes": 69206016, "slow_total_bytes": 320058949632, "slow_used_bytes": 13606322176, "num_files": 220, "log_bytes": 44204032, "log_compactions": 0, "logged_bytes": 31145984, "files_written_wal": 1, "files_written_sst": 1, "bytes_written_wal": 37753489, "bytes_written_sst": 238992 } Even if the new block devices are recognized correctly:2018-11-20 11:40:34.653524 7f70219b8d00 1 bdev(0x5647ea9ce200 /var/lib/ceph/osd/ceph-0/block.db) open size 64424509440 (0xf00000000, 60GiB) block_size 4096 (4KiB) non-rotational 2018-11-20 11:40:34.653532 7f70219b8d00 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block.db size 60GiB2018-11-20 11:40:34.662385 7f70219b8d00 1 bdev(0x5647ea9ce600 /var/lib/ceph/osd/ceph-0/block.wal) open size 2147483648 (0x80000000, 2GiB) block_size 4096 (4KiB) non-rotational 2018-11-20 11:40:34.662406 7f70219b8d00 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-0/block.wal size 2GiBAre we missing some command to "notify" rocksdb about the new device size?All the best, Florian _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
-- EveryWare AG Florian Engelmann Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelmann@xxxxxxxxxxxx web: http://www.everyware.ch
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
