No, it's exactly what I told you it was. "bluestore_bdev_partition_path" is the data path. In all of my scenarios my DB and Data are on the same partition, hence mine are the same. Your DB and WAL are on a different partition from your Data... so your DB partition is different... Whatever your misunderstanding is about where/why your cluster's usage is higher/different than you think it is, it has nothing to do with where your DB and WAL partitions are.
There is a overhead just for having a FS on the disk. In this case that FS is bluestore. You can look at [1] this ML thread from a while ago where I mentioned a brand new cluster with no data in it and the WAL partitions on separate disks that it was using about 1.1GB of data per OSD.
[1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-March/025246.html
On Mon, Oct 22, 2018 at 4:51 PM Robert Stanford <rstanford8896@xxxxxxxxx> wrote:
That's very helpful, thanks. In your first case above your bluefs_db_partition_path and bluestore_bdev_partition path are the same. Though I have a different data and db drive, mine are different. Might this explain something? My root concern is that there is more utilization on the cluster than what's in the pools, the excess equal to about wal size * number of osds...On Mon, Oct 22, 2018 at 3:35 PM David Turner <drakonstein@xxxxxxxxx> wrote:My DB doesn't have a specific partition anywhere, but there's still a symlink for it to the data partition. On my home cluster with all DB, WAL, and Data on the same disk without any partitions specified there is a block symlink but no block.wal symlink.For the cluster with a specific WAL partition, but no DB partition, my OSD paths looks like [1] this. For my cluster with everything on the same disk, my OSD paths look like [2] this. Unless you have a specific path for "bluefs_wal_partition_path" then it's going to find itself on the same partition as the db.[1] $ ceph osd metadata 5 | grep path"bluefs_db_partition_path": "/dev/dm-29","bluefs_wal_partition_path": "/dev/dm-41","bluestore_bdev_partition_path": "/dev/dm-29",[2] $ ceph osd metadata 5 | grep path"bluefs_db_partition_path": "/dev/dm-5","bluestore_bdev_partition_path": "/dev/dm-5",On Mon, Oct 22, 2018 at 4:21 PM Robert Stanford <rstanford8896@xxxxxxxxx> wrote:Let me add, I have no block.wal file (which the docs suggest should be there). http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/On Mon, Oct 22, 2018 at 3:13 PM Robert Stanford <rstanford8896@xxxxxxxxx> wrote:We're out of sync, I think. You have your DB on your data disk so your block.db symlink points to that disk, right? There is however no wal symlink? So how would you verify your WAL actually lived on your NVMe?On Mon, Oct 22, 2018 at 3:07 PM David Turner <drakonstein@xxxxxxxxx> wrote:And by the data disk I mean that I didn't specify a location for the DB partition.On Mon, Oct 22, 2018 at 4:06 PM David Turner <drakonstein@xxxxxxxxx> wrote:Track down where it says they point to? Does it match what you expect? It does for me. I have my DB on my data disk and my WAL on a separate NVMe.On Mon, Oct 22, 2018 at 3:21 PM Robert Stanford <rstanford8896@xxxxxxxxx> wrote:David - is it ensured that wal and db both live where the symlink block.db points? I assumed that was a symlink for the db, but necessarily for the wal, because it can live in a place different than the db.On Mon, Oct 22, 2018 at 2:18 PM David Turner <drakonstein@xxxxxxxxx> wrote:You can always just go to /var/lib/ceph/osd/ceph-{osd-num}/ and look at where the symlinks for block and block.wal point to.On Mon, Oct 22, 2018 at 12:29 PM Robert Stanford <rstanford8896@xxxxxxxxx> wrote:That's what they say, however I did exactly this and my cluster utilization is higher than the total pool utilization by about the number of OSDs * wal size. I want to verify that the wal is on the SSDs too but I've asked here and no one seems to know a way to verify this. Do you?Thank you, R_______________________________________________On Mon, Oct 22, 2018 at 5:22 AM Maged Mokhtar <mmokhtar@xxxxxxxxxxx> wrote:
If you specify a db on ssd and data on hdd and not explicitly specify a
device for wal, wal will be placed on same ssd partition with db.
Placing only wal on ssd or creating separate devices for wal and db are
less common setups.
/Maged
On 22/10/18 09:03, Fyodor Ustinov wrote:
> Hi!
>
> For sharing SSD between WAL and DB what should be placed on SSD? WAL or DB?
>
> ----- Original Message -----
> From: "Maged Mokhtar" <mmokhtar@xxxxxxxxxxx>
> To: "ceph-users" <ceph-users@xxxxxxxxxxxxxx>
> Sent: Saturday, 20 October, 2018 20:05:44
> Subject: Re: Drive for Wal and Db
>
> On 20/10/18 18:57, Robert Stanford wrote:
>
>
>
>
> Our OSDs are BlueStore and are on regular hard drives. Each OSD has a partition on an SSD for its DB. Wal is on the regular hard drives. Should I move the wal to share the SSD with the DB?
>
> Regards
> R
>
>
> _______________________________________________
> ceph-users mailing list [ mailto:ceph-users@xxxxxxxxxxxxxx | ceph-users@xxxxxxxxxxxxxx ] [ http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com | http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ]
>
> you should put wal on the faster device, wal and db could share the same ssd partition,
>
> Maged
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
