Hi Richard,
thanks for all this additional information and testing.
Looking at all of this, deploying OSDs on HDDs with solid state cache devices seems to be a very non-trivial exercise. Depending on how much solid-state capacity you have available, it might very well be best to deploy the OSD with SSD parameters (omitting WAL+DB devices entirely).
I think it would be great if someone with more in-depth knowledge of relevant parameters could offer some general advice in the ceph documentation. Should we open a tracker with a documentation/feature request?
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Richard Bade <hitrich@xxxxxxxxx>
Sent: 11 April 2022 06:38:08
To: Frank Schilder
Cc: Ceph Users
Subject: Re: Ceph Bluestore tweaks for Bcache
Ok, further testing and thinking...
Frank, you mentioned about creating the osd without cache so that it'd
be picked up as HDD not SSD. Also back in this thread Aleksandr
mentioned that the parameter rotational in sysfs is used for this.
So I checked what this parameter is being set to with bcache with and
without a cache disk attached. It's always set to zero. However I also
noticed that since this is in the "virtual" disks section this
parameter is writable.
e.g.
$ echo 1 | sudo tee /sys/devices/virtual/block/bcache0/queue/rotational
1
$ cat /sys/devices/virtual/block/bcache0/queue/rotational
1
So then I did some testing on my test cluster and created an osd on
this bcache disk set to rotational=1. The class gets correctly set to
HDD and when I query the osd metadata I get rotational=1
$ sudo ceph osd metadata 15
--->8---
"bluefs_db_rotational": "0",
"bluefs_db_type": "ssd",
--->8---
"bluestore_bdev_rotational": "1",
"bluestore_bdev_type": "hdd",
--->8---
"devices": "bcache0,md1",
--->8---
"osd_data": "/var/lib/ceph/osd/ceph-15",
"osd_objectstore": "bluestore",
"rotational": "1"
So that's looking pretty good for working around the issue for me.
I'll need to do a bunch more testing as it'll obviously be changing a
few other settings that weren't getting set before on osd creation.
It's an easy add to my process though. After creating bcache device,
set rotational to 1 and carry on as before.
Rich
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
