Hello
We have a new Pacific cluster configured via Cephadm.
For the OSDs, the spec is like this, with the intention for DB and WAL to
be on NVMe:
spec:
data_devices:
rotational: true
db_devices:
model: SSDPE2KE032T8L
filter_logic: AND
objectstore: bluestore
wal_devices:
model: SSDPE2KE032T8L
This was after an initial attempt like this:
spec:
data_devices:
rotational: 1
db_devices:
rotational: 0
based on the suggestion in the docs that this would be sufficient for both
DB and WAL (
https://docs.ceph.com/en/pacific/cephadm/services/osd/#the-simple-case)
ended up with metadata on the HDD data disks, as demonstrated by quite a
lot of space being consumed even with no actual data.
With the new spec, the usage looks more normal. However, it's not clear
whether both DB and WAL are in fact on the faster devices as desired.
Here's an except of one of the new OSDs:
{
"id": 107,
"arch": "x86_64",
"back_iface": "",
"bluefs": "1",
"bluefs_dedicated_db": "0",
"bluefs_dedicated_wal": "1",
"bluefs_single_shared_device": "0",
"bluefs_wal_access_mode": "blk",
"bluefs_wal_block_size": "4096",
"bluefs_wal_dev_node": "/dev/dm-40",
"bluefs_wal_devices": "nvme0n1",
"bluefs_wal_driver": "KernelDevice",
"bluefs_wal_partition_path": "/dev/dm-40",
"bluefs_wal_rotational": "0",
"bluefs_wal_size": "355622453248",
"bluefs_wal_support_discard": "1",
"bluefs_wal_type": "ssd",
"bluestore_bdev_access_mode": "blk",
"bluestore_bdev_block_size": "4096",
"bluestore_bdev_dev_node": "/dev/dm-39",
"bluestore_bdev_devices": "sdr",
"bluestore_bdev_driver": "KernelDevice",
"bluestore_bdev_partition_path": "/dev/dm-39",
"bluestore_bdev_rotational": "1",
"bluestore_bdev_size": "8001561821184",
"bluestore_bdev_support_discard": "0",
"bluestore_bdev_type": "hdd",
"ceph_release": "pacific",
"ceph_version": "ceph version 16.2.7
(dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)",
"ceph_version_short": "16.2.7",
8< >8
"container_image": "
quay.io/ceph/ceph@sha256:a39107f8d3daab4d756eabd6ee1630d1bc7f31eaa76fff41a77fa32d0b903061
",
"cpu": "AMD EPYC 7352 24-Core Processor",
"default_device_class": "hdd",
"device_ids":
"nvme0n1=SSDPE2KE032T8L_PHLN0195002R3P2BGN,sdr=LENOVO_ST8000NM010A_EX_WKD2CHZL0000E02930J6",
"device_paths":
"nvme0n1=/dev/disk/by-path/pci-0000:c1:00.0-nvme-1,sdr=/dev/disk/by-path/pci-0000:41:00.0-scsi-0:0:41:0",
"devices": "nvme0n1,sdr",
"distro": "centos",
"distro_description": "CentOS Stream 8",
"distro_version": "8",
8< >8
"journal_rotational": "0",
"kernel_description": "#1 SMP Thu Feb 10 16:11:23 UTC 2022",
"kernel_version": "4.18.0-365.el8.x86_64",
"mem_swap_kb": "4194300",
"mem_total_kb": "131583928",
"network_numa_unknown_ifaces": "back_iface,front_iface",
"objectstore_numa_nodes": "0",
"objectstore_numa_unknown_devices": "sdr",
"os": "Linux",
"osd_data": "/var/lib/ceph/osd/ceph-107",
"osd_objectstore": "bluestore",
"osdspec_affinity": "dashboard-admin-1645460246886",
"rotational": "1"
}
Note:
"bluefs_dedicated_db": "0",
"bluefs_dedicated_wal": "1",
"bluefs_single_shared_device": "0",
On one of our Nautilus clusters, we have:
"bluefs_single_shared_device": "1",
and the same on an Octopus cluster.
I've heard of the WAL being hosted in the DB, but not the other way
around...
Best Wishes,
Adam
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
