Re: Unclear on metadata config for new Pacific cluster

[Eugen Block <eblock@xxxxxx>]

Hi,

if you want to have DB and WAL on the same device, just don't specify  
WAL in your drivegroup. It will be automatically created on the DB  
device, too. In your case the rotational flag should be enough to  
distinguish between data and DB.

> based on the suggestion in the docs that this would be sufficient for both
> DB and WAL (
> https://docs.ceph.com/en/pacific/cephadm/services/osd/#the-simple-case)
> ended up with metadata on the HDD data disks, as demonstrated by quite a
> lot of space being consumed even with no actual data.

How exactly did you determine that there was actual WAL data on the HDDs?


Zitat von Adam Huffman <adam.huffman.lists@xxxxxxxxx>:

> Hello
>
> We have a new Pacific cluster configured via Cephadm.
>
> For the OSDs, the spec is like this, with the intention for DB and WAL to
> be on NVMe:
>
> spec:
>
>   data_devices:
>
>     rotational: true
>
>   db_devices:
>
>     model: SSDPE2KE032T8L
>
>   filter_logic: AND
>
>   objectstore: bluestore
>
>   wal_devices:
>
>     model: SSDPE2KE032T8L
>
> This was after an initial attempt like this:
>
> spec:
>
>   data_devices:
>
>     rotational: 1
>
>   db_devices:
>
>     rotational: 0
>
> based on the suggestion in the docs that this would be sufficient for both
> DB and WAL (
> https://docs.ceph.com/en/pacific/cephadm/services/osd/#the-simple-case)
> ended up with metadata on the HDD data disks, as demonstrated by quite a
> lot of space being consumed even with no actual data.
>
> With the new spec, the usage looks more normal. However, it's not clear
> whether both DB and WAL are in fact on the faster devices as desired.
>
> Here's an except of one of the new OSDs:
>
>     {
>
>         "id": 107,
>
>         "arch": "x86_64",
>
>         "back_iface": "",
>
>         "bluefs": "1",
>
>         "bluefs_dedicated_db": "0",
>
>         "bluefs_dedicated_wal": "1",
>
>         "bluefs_single_shared_device": "0",
>
>         "bluefs_wal_access_mode": "blk",
>
>         "bluefs_wal_block_size": "4096",
>
>         "bluefs_wal_dev_node": "/dev/dm-40",
>
>         "bluefs_wal_devices": "nvme0n1",
>
>         "bluefs_wal_driver": "KernelDevice",
>
>         "bluefs_wal_partition_path": "/dev/dm-40",
>
>         "bluefs_wal_rotational": "0",
>
>         "bluefs_wal_size": "355622453248",
>
>         "bluefs_wal_support_discard": "1",
>
>         "bluefs_wal_type": "ssd",
>
>         "bluestore_bdev_access_mode": "blk",
>
>         "bluestore_bdev_block_size": "4096",
>
>         "bluestore_bdev_dev_node": "/dev/dm-39",
>
>         "bluestore_bdev_devices": "sdr",
>
>         "bluestore_bdev_driver": "KernelDevice",
>
>         "bluestore_bdev_partition_path": "/dev/dm-39",
>
>         "bluestore_bdev_rotational": "1",
>
>         "bluestore_bdev_size": "8001561821184",
>
>         "bluestore_bdev_support_discard": "0",
>
>         "bluestore_bdev_type": "hdd",
>
>         "ceph_release": "pacific",
>
>         "ceph_version": "ceph version 16.2.7
> (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)",
>
>         "ceph_version_short": "16.2.7",
>
>    8<                                                        >8
>
>         "container_image": "
> quay.io/ceph/ceph@sha256:a39107f8d3daab4d756eabd6ee1630d1bc7f31eaa76fff41a77fa32d0b903061
> ",
>
>         "cpu": "AMD EPYC 7352 24-Core Processor",
>
>         "default_device_class": "hdd",
>
>         "device_ids":
> "nvme0n1=SSDPE2KE032T8L_PHLN0195002R3P2BGN,sdr=LENOVO_ST8000NM010A_EX_WKD2CHZL0000E02930J6",
>
>         "device_paths":
> "nvme0n1=/dev/disk/by-path/pci-0000:c1:00.0-nvme-1,sdr=/dev/disk/by-path/pci-0000:41:00.0-scsi-0:0:41:0",
>
>         "devices": "nvme0n1,sdr",
>
>         "distro": "centos",
>
>         "distro_description": "CentOS Stream 8",
>
>         "distro_version": "8",
>
>     8<                                                       >8
>
>         "journal_rotational": "0",
>
>         "kernel_description": "#1 SMP Thu Feb 10 16:11:23 UTC 2022",
>
>         "kernel_version": "4.18.0-365.el8.x86_64",
>
>         "mem_swap_kb": "4194300",
>
>         "mem_total_kb": "131583928",
>
>         "network_numa_unknown_ifaces": "back_iface,front_iface",
>
>         "objectstore_numa_nodes": "0",
>
>         "objectstore_numa_unknown_devices": "sdr",
>
>         "os": "Linux",
>
>         "osd_data": "/var/lib/ceph/osd/ceph-107",
>
>         "osd_objectstore": "bluestore",
>
>         "osdspec_affinity": "dashboard-admin-1645460246886",
>
>         "rotational": "1"
>
>     }
>
> Note:
>
>        "bluefs_dedicated_db": "0",
>
>        "bluefs_dedicated_wal": "1",
>
>        "bluefs_single_shared_device": "0",
>
> On one of our Nautilus clusters, we have:
>
> "bluefs_single_shared_device": "1",
>
> and the same on an Octopus cluster.
>
> I've heard of the WAL being hosted in the DB, but not the other way
> around...
>
> Best Wishes,
> Adam
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx



_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx